Gossamer Forum
Home : Products : Gossamer Links : Development, Plugins and Globals :

Re: Metacrawler Plugin - Will Pay

Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Hi,

The spider is a full featured spider. It will go out and spider any URL's you give it (with hooks to easily spider your Links SQL entries). It visits each page, indexes it and queues all the links on those pages to be spidered later.

It hooks into Links SQL by allowing your visitors a better search. Instead of just searching on the URL/Title/Description of a site, you can search on content found on the site. For instance MySQLMan may not be mentioned in a listing about Gossamer Threads, but if the site had been spidered, you'd be able to find it.

That's a simplified view, it is a full featured spider and:

* obeys robots.txt file
* can enable/disable spidering remote sites
* can set filters/rules on whether content should be indexed, or spidered
* can import links into Links SQL directory
* can enhance search, or offer second search (i.e. search directory/search the web).
* can spider any urls, not just your directory sites.

Let me know if that makes sense..

Cheers,

Alex

--
Gossamer Threads Inc.
Subject Author Views Date
Thread Metacrawler Plugin - Will Pay robfromaaa 5067 Feb 20, 2001, 3:50 PM
Thread Re: Metacrawler Plugin - Will Pay
Paul 5048 Feb 20, 2001, 3:55 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5040 Feb 20, 2001, 4:08 PM
Thread Re: Metacrawler Plugin - Will Pay
Paul 5032 Feb 20, 2001, 4:13 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 4993 Feb 20, 2001, 4:16 PM
Post Re: Metacrawler Plugin - Will Pay
Paul 5018 Feb 20, 2001, 4:17 PM
Thread Re: Metacrawler Plugin - Will Pay
Alex 5028 Feb 20, 2001, 4:55 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5004 Feb 20, 2001, 5:58 PM
Thread Re: Metacrawler Plugin - Will Pay
Alex 5040 Feb 20, 2001, 6:36 PM
Post Re: Metacrawler Plugin - Will Pay
robfromaaa 5014 Feb 20, 2001, 7:16 PM
Thread Re: Metacrawler Plugin - Will Pay
robfromaaa 5000 Feb 21, 2001, 1:26 AM
Thread Re: Metacrawler Plugin - Will Pay
Robert_B 5000 Feb 21, 2001, 4:07 AM
Post Re: Metacrawler Plugin - Will Pay
padders 4929 Feb 23, 2001, 8:36 AM