Alex,
As you KNOW, I've been salivating for this spider for a long time. Here are my opening questions (hopefully this will be all of them prior to taking the plunge):
1) When I initially migrated from my previous engine, I imported from a flatfile the URL, Title and Description, even though that flatfile also contains Categories and Keyword meta data that I would have liked to import as well at that time. When I imported and reindexed, NG autoassigned ID numbers to each link. I now have 13,964 fully searchable links in the MAIN database even though categories have been assigned to only 509 of them. Searches point to the links being contained in the Home category even though Home is not browsable (Nor do I care to make it so). But it does beg the question, why can't I add to the Home category? Suppose someone wants to add a link to the general database that doesn't fit into a specific category. Importing from MySQLMan does it automatically. How can one do this manually? Would the spider be able to?
2) I have since added a Keyword column to the Links table, updated the def files and assigned it a weight for searching together with URL, Title and Description, even though Keywords will not be displayed on the search results page.
a) Can the spider gather Keyword meta data from pages into a separate column in the spider database?
b) Can the spider build from a pipe delimited text flatfile? or would it just use the URLs from that file to go out and spider those pages?
c) If it can build directly from a flatfile and it contains URL|Title|Category|Description|Keywords, can the spider make it fairly simple to add records to the main database complete with Categories and Keywords? (Remember, Category will often be blank so those records will need to associate with the Home category when transferring to the main database.) I tried this initially from MySQLMan but couldn't figure it out because Links and Catlinks are separate tables. Importing simultaneously to both or synchronizing them afterward is where I get lost. If there's a way to do it here I'd still like to know how but it seems to me the spider might handle it better.
3) Can the spider be used as the manual submission engine so that it prevalidates manual entries, leaving them in the spider database until validated and transferred to the main? I'm still using my old spider to prevalidate, dump to a flatfile and import later in MySQLMan. It's cumbersome at best, and it does not assign categories. I then have to go back in, search the URLs and assign categories manually. Ugh.
Bottom line, do you think this spider is my ticket?
Mark Brasche
http://SurfSafely.com/
As you KNOW, I've been salivating for this spider for a long time. Here are my opening questions (hopefully this will be all of them prior to taking the plunge):
1) When I initially migrated from my previous engine, I imported from a flatfile the URL, Title and Description, even though that flatfile also contains Categories and Keyword meta data that I would have liked to import as well at that time. When I imported and reindexed, NG autoassigned ID numbers to each link. I now have 13,964 fully searchable links in the MAIN database even though categories have been assigned to only 509 of them. Searches point to the links being contained in the Home category even though Home is not browsable (Nor do I care to make it so). But it does beg the question, why can't I add to the Home category? Suppose someone wants to add a link to the general database that doesn't fit into a specific category. Importing from MySQLMan does it automatically. How can one do this manually? Would the spider be able to?
2) I have since added a Keyword column to the Links table, updated the def files and assigned it a weight for searching together with URL, Title and Description, even though Keywords will not be displayed on the search results page.
a) Can the spider gather Keyword meta data from pages into a separate column in the spider database?
b) Can the spider build from a pipe delimited text flatfile? or would it just use the URLs from that file to go out and spider those pages?
c) If it can build directly from a flatfile and it contains URL|Title|Category|Description|Keywords, can the spider make it fairly simple to add records to the main database complete with Categories and Keywords? (Remember, Category will often be blank so those records will need to associate with the Home category when transferring to the main database.) I tried this initially from MySQLMan but couldn't figure it out because Links and Catlinks are separate tables. Importing simultaneously to both or synchronizing them afterward is where I get lost. If there's a way to do it here I'd still like to know how but it seems to me the spider might handle it better.
3) Can the spider be used as the manual submission engine so that it prevalidates manual entries, leaving them in the spider database until validated and transferred to the main? I'm still using my old spider to prevalidate, dump to a flatfile and import later in MySQLMan. It's cumbersome at best, and it does not assign categories. I then have to go back in, search the URLs and assign categories manually. Ugh.
Bottom line, do you think this spider is my ticket?
Mark Brasche
http://SurfSafely.com/
More spider questions