Gossamer Forum
Home : Products : Gossamer Links : Development, Plugins and Globals :

Re: Metacrawler Plugin - Will Pay

Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Hi,

The spider is a full featured spider. It will go out and spider any URL's you give it (with hooks to easily spider your Links SQL entries). It visits each page, indexes it and queues all the links on those pages to be spidered later.

It hooks into Links SQL by allowing your visitors a better search. Instead of just searching on the URL/Title/Description of a site, you can search on content found on the site. For instance MySQLMan may not be mentioned in a listing about Gossamer Threads, but if the site had been spidered, you'd be able to find it.

That's a simplified view, it is a full featured spider and:

* obeys robots.txt file
* can enable/disable spidering remote sites
* can set filters/rules on whether content should be indexed, or spidered
* can import links into Links SQL directory
* can enhance search, or offer second search (i.e. search directory/search the web).
* can spider any urls, not just your directory sites.

Let me know if that makes sense..

Cheers,

Alex

--
Gossamer Threads Inc.
Subject Author Views Date
Thread Metacrawler Plugin - Will Pay robfromaaa 5114 Feb 20, 2001, 3:50 PM
Thread Re: Metacrawler Plugin - Will Pay
Paul 5095 Feb 20, 2001, 3:55 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5088 Feb 20, 2001, 4:08 PM
Thread Re: Metacrawler Plugin - Will Pay
Paul 5080 Feb 20, 2001, 4:13 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5040 Feb 20, 2001, 4:16 PM
Post Re: Metacrawler Plugin - Will Pay
Paul 5065 Feb 20, 2001, 4:17 PM
Thread Re: Metacrawler Plugin - Will Pay
Alex 5076 Feb 20, 2001, 4:55 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5051 Feb 20, 2001, 5:58 PM
Thread Re: Metacrawler Plugin - Will Pay
Alex 5088 Feb 20, 2001, 6:36 PM
Post Re: Metacrawler Plugin - Will Pay
robfromaaa 5062 Feb 20, 2001, 7:16 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5047 Feb 21, 2001, 1:26 AM
Thread Re: Metacrawler Plugin - Will Pay
Robert_B 5048 Feb 21, 2001, 4:07 AM
Post Re: Metacrawler Plugin - Will Pay
padders 4976 Feb 23, 2001, 8:36 AM