Hi!

i have carefully read manual at http://www.gossamer-threads.com/perl/plugins/help.cgi?do=help&topic=Spider/help.html but i still do have a question.
Does spider enters validated links into SQL data i mean, why are there in the search results templates checkboxes for spider databse .. that measn i have two databases?

Gregor

Hi Gregor,

Yes, in essence there are two databases.

The reason there are two databases is simple: Volume. The number of pages the spider can download can out number the number of pages in your links by an immense amount. While the Links database holds pointers to the initial pages of a website, the spider will hold in its database virtually all accessible pages in that site. So putting the spidered links into one of your Links database will clutter up your categories with needless junk.

Still, when you download a webpage off the internet, you don't know if the page is totally unrelated to the content you want on your site. (such as a family oriented portal and the spider downloads stuff of an adult site). The validation allows you to first check what sort of information was downloaded.

Still, you can choose to automatically make a downloaded page available using the "Auto Validate" option. But with the above caveat. ;)

Hope that helps,
Aki

thanx for explanation :)
bye

Gregor