Gossamer Forum
Home : Products : Gossamer Links : Discussions :

Spider Shutdown?

Quote Reply
Spider Shutdown?
Hi there, I upgraded to LinkSQL a couple of days ago and everything is running like a dream. But I also got the spider plugin and that is a different matter.

I have read a couple of posts about the spider shutdown problems but nobody seemed to have posted any solutions.

What the Problem Is:-

I have started Spider to do a search for links and it keeps going and going with no changes to the status screen.
When I press SHUTDOWN nothing happeds, it just keeps going.
When I press PAUSE still nothing happens, it just keeps on going.
I have left it for hours and hours before pressing shutdown and after having pressed it.

Is there some other way to shut the spider off as I don't want it to keep going forever?

Tom Martin
Quote Reply
Re: Spider Shutdown? In reply to
Another way to stop the spider is

kill -9 <process id>

;-)



-------------------
Heiko Mentzel
http://findgay.net/
-------------------
Quote Reply
Re: [tommartin] Spider Shutdown? In reply to
Say, I have this problem too. And nothing shows up in ps -ef |grep Spider.pl

Anybody else with this problem? This is a show stopper.
Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
I can't get this spider to stop!

I just finished rebooting the machine.

My spider status screen shows it still active, the constant clicking as it refreshes seems to confirm this although there is nothing in ps -ef |grep Spider.pl


Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
I see why there is nothing in ps -ef |grep Spider.pl, it is because as soon as you type it and look the process has stopped. If you keep typing it over again, you'll see a PID for it. You have to be quick.

Anyway, so the Spider.pl process is also confirmed to be running through ps -ef.

Now what?

I clicked on Pause, and waited a bit and still found the process on ps -ef.

I clicked on Shutdown, and waited a bit and also still find the process on ps -ef.

Any help? This is amazing that not even a reboot killed it.

As far as I can tell it isn't even adding links as the added column remains at 0 on the status display.


Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
Oh, BTW, the PID constantly changes so I can't KILL it.

By the time you ps -ef, continuously, find the PID, type the PID in, it comes back and says "No such PID".
Quote Reply
Re: [tommartin] Spider Shutdown? In reply to
Hi Tom,

If you running a linux system, what happens if you do a

ps ax | grep spider.pl

Does it show up in the process list?

Aki
Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
Whoa, that's wierd.

The spider is not the infernal critter that won't die. It shouldn't be showing up on the process list after a full reboot.

If possible, can you send me login information to your server so I can take a look at it?

Aki
aki@gossamer-threads.com
Quote Reply
Re: [Aki] Spider Shutdown? In reply to
Aki,

I just sent you the info.

Thanks for the quick reply.

John
Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
Yes, it shows up with:

ps ax | grep spider.pl

John


Quote Reply
Re: [takacsj] Spider Shutdown? In reply to
Well, I finally got the spider to stop.

I rebooted again to no avail.

I checked various startup files to try and discover why the spider would continue spidering after a reboot, to no avail.

Finally in my frustration, I kicked off multiple spider instances, and instantly the spider stopped!

So after several hours of spider an HTML page with several thousand links, it has absolutely zero links to validate.

I've spidered sites individually, no problem, but it couldn't spider this monster HTML page with several thousand links for some reason.

By the way, I have 10's of thousands of links in a CSV file. It would be nice if I could import that into the spider for queing.

John