Gossamer Forum
Quote Reply
[ UPDATE ] Spider v2.0
Spider

After a long overdue amount of time, since the last major update (2004) - I've now done a major re-vamp of the Spider plugin. These are mainly cosmetic, and speed improvments. No real new features yet, as this is more of a "cleanup" version, using CSS formatting, and a MUCH more detailed "Readme".

The idea behind this plugin is to spider another site for links. It will go through the page you enter (currently only one depth), and grab all the links on that page. Now works on LinksSQL 2.1 + and Gossamer Links 3.x + .

PRICE: $50 [ details ]

Screenshots: Please see this page: http://www.ultranerds.com/Products/Plugins/Links_SQL_Plugins/Spider_L20/

Plugin Description:

The idea behind this plugin is to spider another site for links. It will go through the page you enter (currently only one depth), and grab all the links on that page. You can then select which pages you want to spider. Once you decide, then you continue, and it will go to each of the pages, and grab the requested details (description, title and author).

Its all done by meta-tag recognition, so its not fool proof by any means. I got about a 95% success rate with it accurately grabbing the details. You are then shown a list of all the sites that were spidered. It shows the details that were grabbed, and gives you a direct link to add the form (via admin, with all forms pre-completed). Also, inline checking to see if the URL exists in the directory already.

Now works on LinksSQL 2.1 + and Gossamer Links 3.x +

Plugin Requirements:
  • Links SQL 2.1+L
  • The Following Perl Modules
    • CGI.pm
    • LWP::Simple
    • LWP::UserAgent
    • HTML::LinkExtor
    • URI::URL
Features:
  • Ability to turn on/off the Whois checks... NEW
  • 2 different types of category dropdown boxes (1 takes longer to generate, but is better on the eyes ... this can be turned on/off as required in the plugins settings) NEW
  • Spiders a site, and grab a list of URL's from it...
  • CSS Based interface, making the pages cleaner and easier to use (making use of the <label..> feature). NEW
  • Grabs a sites URL, title, description and author (or allows you to use predefined values)
  • Checks if the URL is already in your database, and if it is, reports a prominent error to you, so you know about it...
  • Other things, which I'll add once I remember them :p

If you have any questions, please don't hestiate to ask.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Last edited by:

Andy: Apr 19, 2007, 9:25 AM
Quote Reply
Re: [Andy] [ UPDATE ] Spider v2.0 In reply to
Hi,

Just a note, to let people know Spider v2.1 is now out (for GLinks 3.0.1, 3.0.4, 3.20+)

Its nothing major - but it just includes the latest version of Net::Whois, for doing Whois checks on the pages you are spidering (i.e for getting ther site owner details, etc - if they can't be found in the meta-tags).

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!