Hi,

The spider is a full featured spider. It will go out and spider any URL's you give it (with hooks to easily spider your Links SQL entries). It visits each page, indexes it and queues all the links on those pages to be spidered later.

It hooks into Links SQL by allowing your visitors a better search. Instead of just searching on the URL/Title/Description of a site, you can search on content found on the site. For instance MySQLMan may not be mentioned in a listing about Gossamer Threads, but if the site had been spidered, you'd be able to find it.

That's a simplified view, it is a full featured spider and:

* obeys robots.txt file
* can enable/disable spidering remote sites
* can set filters/rules on whether content should be indexed, or spidered
* can import links into Links SQL directory
* can enhance search, or offer second search (i.e. search directory/search the web).
* can spider any urls, not just your directory sites.

Let me know if that makes sense..

Cheers,

Alex

--
Gossamer Threads Inc.