Gossamer Forum
Quote Reply
I need a Spider plug-in
Please excuse me if this question has already been answered :) but I just can't seem to find enough details before I purchase.

I have LinksSQL on one site and about 200 .html pages on another site which contain a link and a 30 word description. These are all hand created pages and were never obtained from DMOZ.

Questions:

1. Which spider plug-in product (GT or Andy or others) will grab/spider the link and the complete description and place same in the cue for me to approve?

2. Does this spider run from shell with a series of commands or is it broswer based?

3. Can I spider a Cat/sub cat in my 200 pages and then tell the little spider to approve and insert in my matching LinksSQL Cat/sub cat?

Thanks in advance for your help. :)

geek/talk Forums @ GeekVillage.com
Quality Forums For Webmasters & Merchants

Quote Reply
Re: [Steve_S] I need a Spider plug-in In reply to
Please PM, and I'll install it on your server for a test, so you can see if mine will do the job for you.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Steve_S] I need a Spider plug-in In reply to
I don't mean to diss Andy's plugin but you'll have much more power and flexibility with GT's spider, the only thing is the price. But if you can afford it I would go with it. It runs as a daemon and you can create rules that the spider follows.
Quote Reply
Re: [Paul] I need a Spider plug-in In reply to
i'm going to come to andy's defense here and say that i use his spider and it has been very useful in increasing productivity.

Now I am a complete doofus when it comes to programming. I've asked GT about their spider and have been told on many occasions that its difficult to set up and run. In other words, I shouldn't buy itTongue

The guys at GT tell me that they were working a newer more user friendly version but that they've been too busy and that further work on an update is stalled indefinately.

I HOPE THEY CAN GET A NEW VERSION OUT SOON! THAT'S A HINT GT!Smile

In the meantime, Andy's spider does the job quite well.

I have no idea how difficult it would be to develop a user friendly(ie doofus proof) spider. If you did one tho Paul, I'd certainly almost buy it.
Quote Reply
Re: [archer] I need a Spider plug-in In reply to
Thanks everyone and happy Turkey Day :)

I'm still confused so please indulge me.

My questions aren't answered. Will either spider or others do exactly what I need to accomplish? Please see my original questions. Yes or no please.

OK. One questions is answered. They both use shell/dameon.

Thanks in advance.

geek/talk Forums @ GeekVillage.com
Quality Forums For Webmasters & Merchants

Last edited by:

Steve_S: Nov 28, 2002, 9:38 AM
Quote Reply
Re: [Steve_S] I need a Spider plug-in In reply to
Hi...here we go;

1. Which spider plug-in product (GT or Andy or others) will grab/spider the link and the complete description and place same in the cue for me to approve? Mine has an 'add' link to click after spidering, which then takes you to a pre-populated add form for the links, ready for you to choose a category, and complete any details that were not grabbed.

2. Does this spider run from shell with a series of commands or is it broswer based? Mine runs via the browser, and not Shell.

3. Can I spider a Cat/sub cat in my 200 pages and then tell the little spider to approve and insert in my matching LinksSQL Cat/sub cat? My one will spider the URL you give it, and then grab any links on that page, ready for you too select. You then choose which of the links you want to spider, and then my spider will goto each of these sites, and send you back the URL/Title/Description and Author.

Hope that answers your questions Smile

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] I need a Spider plug-in In reply to
Quote:
Mine runs via the browser, and not Shell.

That is a major disadvantage and really limits it's flexibility and speed.
Quote Reply
Re: [Paul] I need a Spider plug-in In reply to
I know....but the reason I started writing this plugin, was because a client wanted to be able to grab links from other sites, and then manually assign categories to them. For that reason, I didn't build in SSH/Cron running. I don't really understand where the links would go if they were imported via the SSH/Cron via the Spider, as an appropriate category could not be assigned to them Unsure

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] I need a Spider plug-in In reply to
Thanks a bunch. Please, if I'm the first person on planet earth to ever wan't to grab my descriptions which are far more exstensive than some Meta tag, please just tell me so. :)

Will the GT spider grab my descriptions? Will any persons spider on planet earth grab my descriptions?

Respectfully and with a great deal of admiration, the ability to migrate/spider all the data from your .html pages is a major sales tool.

Then again, I'm not writing the code :)

geek/talk Forums @ GeekVillage.com
Quality Forums For Webmasters & Merchants

Quote Reply
Re: [Steve_S] I need a Spider plug-in In reply to
You can read about the plugin here:

http://gossamer-threads.com/scripts/plugins/spider/