Gossamer Forum
Quote Reply
[Upcoming] Spider_It

Price: $50

Available: Hopefully end of the week.

Idea: The idea behind this plugin is to spider another site for links. It will go through the page you enter (currently only one depth), and grab all the links on that page. You can then select which pages you want to spider. Once you decide, then you continue, and it will goto each of the pages, and grab the requested details (description, title and author). Its all done by meta-tag recognision, so its not fool proof by any means. I got about a 95% success rate with it accuratly grabbing the details. You are then shown a list of all the sites that were spidered. It shows the details that were grabbed, and gives you a direct link to add the form (via admin, with all forms pre-completed). Also, inline checking to see if the URL exists in the directory already.

Current features:

  • You select a URL, and it spiders it for URL's.
  • Can save hundreds of hours going to each new site you want to add, and getting the meta-tag descriptions, or making up your own.
  • VERY Easy to use. Simply enter the URL, and your on your way...Smile

Limitations:

  • If the site you are grabbing the links from, is a LSQL/Links 2 directory site, then the spider cannot get the URL's correctly, so it has to list jump.cgi. This can be worked out by simply clicking on the title of the spidered page, and then just copying and pasting the URL from your browser.
  • I'll let you know once I find any more ;)

If you have any ideas on this, please let me know.

Cheers



Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Last edited by:

Andy: Nov 7, 2002, 2:11 AM
Subject Author Views Date
Post [Upcoming] Spider_It Andy 1450 Nov 7, 2002, 1:55 AM