Spider Plugin,
Ive looked on both the gossamer spider and Andy's plugin spider. But I can really confirm if it can do what I need it for.
Im wondering if any of the two or any other plugin can do the following.
Spider, index and add links to the database.. by Admin / Myself giving the spider a URL, for example http://dir.yahoo.com/...d_Internet/Contests/ which contains the links im interested in. The spider visit the URL and pull all the URL from that page, go to those pages (no further) and pull their titles, description etc and add them into my database for me to validate them and also select a category for them.
What Im trying to do is to simply be able to add links with their description from other sites to my own database. Much like google or yahoo does..
The existing spiders does not seem to able to do this, am I wrong? It seems as a fairly simple thing to do but there isnt any automatic way to do it, the closest thing I can see is a DMOZ import, but im not willing to download a 2gig file to my server etc..
Ive looked on both the gossamer spider and Andy's plugin spider. But I can really confirm if it can do what I need it for.
Im wondering if any of the two or any other plugin can do the following.
Spider, index and add links to the database.. by Admin / Myself giving the spider a URL, for example http://dir.yahoo.com/...d_Internet/Contests/ which contains the links im interested in. The spider visit the URL and pull all the URL from that page, go to those pages (no further) and pull their titles, description etc and add them into my database for me to validate them and also select a category for them.
What Im trying to do is to simply be able to add links with their description from other sites to my own database. Much like google or yahoo does..
The existing spiders does not seem to able to do this, am I wrong? It seems as a fairly simple thing to do but there isnt any automatic way to do it, the closest thing I can see is a DMOZ import, but im not willing to download a 2gig file to my server etc..