Gossamer Forum
Home : Products : Gossamer Links : Pre Sales :

Spider Plug In for LSQL 2.1.0

Quote Reply
Spider Plug In for LSQL 2.1.0
I notice a requirement for the Spider Plug-In (Gossamer Threads) is "Please note: Shell access is required for this plugin, as you must start the plugin from shell."

Does this mean that shell/telnet is required only for the inital installation of the plug-in, and all of its functions can be then run without shell access (I do not have shell or telnet access with my host)? Or does it mean that to be able to use the spider plug-in's fuctions I require on-going shell/telnet access?

ThanksSmile
Quote Reply
Re: [sooke] Spider Plug In for LSQL 2.1.0 In reply to
I believe it is only to initially set up the cron etc.

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [A.J.] Spider Plug In for LSQL 2.1.0 In reply to
Hmm I don't think it uses cron Andy Wink

It runs as a daemon.

>>
Does this mean that shell/telnet is required only for the inital installation of the plug-in, and all of its functions can be then run without shell access (I do not have shell or telnet access with my host)? Or does it mean that to be able to use the spider plug-in's fuctions I require on-going shell/telnet access?
<<

You need telnet access to start the spider, all other stuff is done from the plugins menu.

Last edited by:

Paul: Apr 9, 2002, 5:16 AM
Quote Reply
Re: [Paul] Spider Plug In for LSQL 2.1.0 In reply to
Thanks for replying guys, but I am still a little worried that if I get this plug-in I may not be able to use it.

For the unix ignorant here, is cron somthing that my host must support, or is it something I can just write myself and install?

In the Build menu of LSQL there is FileMan.... it seems I may be able to run command from within this.... I wonder of the spider can be used and started fully fromo with in this, without the shell access?
Quote Reply
Re: [sooke] Spider Plug In for LSQL 2.1.0 In reply to
Hi There,

Cron can be used to launch the spider on a daily basis, however, you can just as well use the file man command function to start it. When the spider first came out, fileman's ability to run programs was not fully developped and even now, it's nice to have shell access if you're going to be using the spider. (just so that you can use ps ax and other tools monitor it's performance.... things such as server load)

One thing I have noticed about the spider is due to my attempt at writing an almost-multithreaded app in a single thread (perl threads are somewhat unstable and not consistant across versions)... if anything locks up in the spider, the entire spider is locked up. So sometimes when the spider is trying to contact a non-existant server, it will lock up and seem to ignore all requests for the spider to shutdown or restart. In this situation, it's wonderful to have shell access so you can force terminate the spider or take a look at what's happening easier.

Hope this helps,

Aki
Quote Reply
Re: [Aki] Spider Plug In for LSQL 2.1.0 In reply to
Thanks, yes this was a help.

The last thing I want to do is lock up my host's server!

I am undecided about getting the spider plug-in or not now.... in my situation without the shell access I could run into plenty of probs.... which is too bad, as it sounds like a really useful plug-in!

Thanks again for the info!
Quote Reply
Re: [sooke] Spider Plug In for LSQL 2.1.0 In reply to
Whoop! Just a little clarification Wink It's not going to lock up your host's server, just the spider. Quite different, the spider runs on its own without the help of the server (I guess it kind of becomes a mini-server of it's own but only listens to you. It should not interact with any of the other systems either. I still wouldn't run it as root, however)

About getting the spider, how would you like to use it?
Quote Reply
Re: [Aki] Spider Plug In for LSQL 2.1.0 In reply to
Hmm, got you now (re locking up server).

I currently use DMOZ extractor, LINKS III, and have played around with Innerprise URO Spider Pro.

All have served their purposes, but lack built in conveinence. I run a Canadian Only directory so-

My particular requirement for a Spider is thus:
  • Easily add small to large amounts of links from any page with the ability to seperate non-Canadian Links.... this sounds difficult, but a lot of Canadian Links these days have the .ca extension. Or if I am grabbing links from a yahoo.ca subdirectory for example... I know that the links are already all Canadian. At the moment , I have to extract the links, then perform a file converision, then import... three steps... i'd like this to be one step! I do not blanket spider entire sites... I find a page which has say 20-500 links which are relevant to one particular category... ie Canada->Arts->Music->Artisits & Bands->Jazz... and extract from here...
  • Fill in and refresh meta fields on existing links i.e. update or add missing description, title etc


LSQL 2.1 already has the ability to check links... so I am assuming this would stay the same with the addition of the spider?

Thanks (I hope this actually answered your question).

Last edited by:

sooke: Apr 9, 2002, 4:01 PM
Quote Reply
Re: [sooke] Spider Plug In for LSQL 2.1.0 In reply to
Hmm...

So in a nutshell, (assuming I'm hearing you right) it's to import the links from say yahoo.ca into your database. Unfortunately the spider may not be what you're looking for, it analyses the content of the pages but it doesn't "quite" fit in that category where it take urls off of directories like yahoo. The spider is designed to operate similiar to altavista or google -Simple interface with the ability to search within a large collection of pages and pull up the pages that best suit your search criteria.

This isn't to say that we can't do anything for you, I'd suggest you send a quick email to Jack, jack@gossamer-threads.com with a url to this discussion and I'm sure he can help you out.

Hope this helps :)

Aki
Quote Reply
Re: [Aki] Spider Plug In for LSQL 2.1.0 In reply to
Ok, will do... thanks!
Quote Reply
Re: [Aki] Spider Plug In for LSQL 2.1.0 In reply to
Was any such spider ever created? I would be interested in a spider / plugin that does exactly that - Easily import links with their titles and description from specified URL..

Thanks
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
There are at least two that I know of
Gossamer Threads hass one here.

http://www.gossamer-threads.com/scripts/links-sql/plugins/spider/

Andy from Ultranerds also has one

http://www.ultranerds.com/cgi-bin/details/20.html

Hope this helps
ccunet
my Christian web
Quote Reply
Re: [ccunet] Spider Plug In for LSQL 2.1.0 In reply to
Thanks, yes I have seen those two plugins. But as described further up in this post they dont actually do what I need them for. The user who asked for a spider plugin, like myself need something that can spider given pages and pull off titles, url and description and add them do the database.. In short.

Thanks anyway.
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
In Reply To:
Thanks, yes I have seen those two plugins. But as described further up in this post they dont actually do what I need them for. The user who asked for a spider plugin, like myself need something that can spider given pages and pull off titles, url and description and add them do the database.. In short.

Thanks anyway.

Which is what mine does Wink

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
In Reply To:
like myself need something that can spider given pages and pull off titles, url and description and add them do the database.. In short.

Thanks anyway.

I don't know much about the gossamer spider. But the one from ultra nerds does what described. Just today I spidered several links from single URL and added them to LSQL. I got the Title URL description and email. YOu can set the depth you want to spider the page. The latest version can also be configured to check whois for valid info.

The process is not fully automated ( you have to manually click to add to the database) and only spiders one page at a time so if that is what you are looking for I misunderstood.

There is also the DMOZ wizard which does bulk imports from DMOZ directory.
ccunet
my Christian web
Quote Reply
Re: [ccunet] Spider Plug In for LSQL 2.1.0 In reply to
Thanks for your reply. Well then maybe it is what I need then.

So you can basically specify a URL, the spider then pick up any links on that page and add them with their title and description to the datebase? Rather than only the description and title of the page which you specified the URL for..

I hope it works like that, because then I could certainly use it.

Thanks!
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
You are correct! That is just what it does. Smile

ccunet
my Christian web
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
If you want to give it a go.. let me know :) I've done enough work for you in the past to trust that you will pay, if you decide to use it. PM if you would like a copy :)

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Spider Plug In for LSQL 2.1.0 In reply to
Andy,

I already got your Ultra package so Ive got the plugin already. Thanks!

Tried it out today and it works very well. Its actually what I need.

Thanks again.

Bjorn
Quote Reply
Re: [Andy] Spider Plug In for LSQL 2.1.0 In reply to
One major thing though..

I suddenly now get the following error when people try and signup through community:

Unable to load Gossamer List because: Bareword "MOD_PERL" not allowed while "strict subs" in use at /home/demon/gaydemon.com/cgi-bin/list/private/lib/GList.pm line 220. Bareword "SPEEDY" not allowed while "strict subs" in use at /home/demon/gaydemon.com/cgi-bin/list/private/lib/GList.pm line 221. Compilation failed in require at /home/demon/gaydemon.com/cgi-bin/community/private/lib/Community/Apps/Local/GList.pm line 223.

Which as you can understand is VERY bad.. I have not got ANY idea what it means...
Quote Reply
Re: [demon] Spider Plug In for LSQL 2.1.0 In reply to
Man... its gonna be a "great" day today, I can just tell Frown

That won't have anything to do with the plugin installation. All it does, is install the plugin, and then call a few perl routines in the plugin itself. It doesn't have any interaction with the other scripts whatsoever (even when you add the link it still has to be done by you).

Sorry to sound so grouchy... but I've just worken up to some not-so good news.. and I'm a little bit pissed off Frown (nothing to do with this post, just generally not in a good mood).

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!