When gathering information from a page, the spider has no way of knowing what category to assign to that page. Therefore, by default the spider database probably has no "Category" column. Categories are assigned to records at the time they are transferred to the Links database, either individually or through a bulk operation.

I will want to add a category column to the spider database even though most of the time it will be blank. The times it will not be blank is when someone manually submits a link to my directory through a form that sends it to the spider and specifies the category in a form field at the time of submission. When transferring to the Links database, I want those records that have already been assigned a category to automatically be placed into that Links category during a bulk transfer operation; Spider database records that have not been assigned a category to automatically go into the Home directory, in the exact same fashion as it does when Importing records through MySQLMan.

I think the best way to understand what I want out of the spider is to try submitting some pages which you've authored to my index as it stands right now. Sites that pass the spider's rules dump to one flatfile; Sites that fail to another flatfile. The URL for my submission page is http://surfsafely.com/urladd.html

I'll gladly do the grunt work as long as I know it's not an impossibility, and who better to ask than the author? What do you think? Will this be doable with your spider?

Mark Brasche
http://SurfSafely.com/