Gossamer Forum
Quote Reply
Spider 2.0
Hi!

Aki has just finished a major update to the Spider which should fix a lot of the problems people were having. I'm sorry about how long this took to get done! Please feel free to try it out and see how it works for you. If you run into any problems, post them here and I'll make sure it gets fixed asap.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Spider 2.0 In reply to
Hi!
Running the spider I ran into problems:
1. The spider is adding links to the queue, but none of the links is available for validation.
2. The spider will not shutdown when I press the "Shutdown" link.
3. I started spider again from telnet, and now I have got two spiders running simultaneusly, none will stop.
4. The Help section for the "spider status" explains a lot of parameters thet is not shown in "spider status"

My spider status looks like this:

Spider is currently running with the following parameters
added 0
new_hosts 4
queued 5216
rejected 0
total_Hosts 1
total_Queue 100
total_Rejected 4
total_links 0

Pause Restart Shutdown

This was after spidering in about 24 hours only one site. Why are there no links to walidate or in the spider-database? I search the database with queue - list all, then I get results, but no links to validate.
Quote Reply
Re: Spider 2.0 In reply to
I'm having some similar issues.

I had to start the spider as "root" (I think because my userID did not have access to the database the spider was trying to run as). Because it was "root", perhaps there was no way for the program to cause it to stop, although the password should allow the program to stop from the inside.

The spider did not shutdown from the admin panel, or if it did, the delay was so long, it was not possible to determine that it really did. There should be some extra status message "shutting down...." or "restarting...." if the command was accepted, and child processes were waiting to die off.

I never got two spiders, but I did have a problem starting the spider. I tried it both with and without "&", and depending on how I was running it, it worked both ways, (or not both ways, depending on how you were looking at it).

added 0
new_hosts 4
queued 5216
rejected 0
total_Hosts 1
total_Queue 100
total_Rejected 4
total_links 0

I would like a better explanation of what each of those parameters is, and what is going on with the insertions to the database.

I actually got more spidering activity with this version, than with previous ones. There are still some issues, most of which are probably related to not really knowing how to use/run the spider properly (as intended), before trying to make it do other stuff :)

PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Re: Spider 2.0 In reply to
I get the following error when trying to run the new spider.pl.

"> ./spider.pl
Checking for another instance of the spider
Starting Spider
> Spider Daemonized
Permission denied at ./spider.pl line 285."


Wilhelm Eilertsen


Quote Reply
Re: Spider 2.0 In reply to
This has to do with the user you are trying to start the spider as, "you" doesn't have access to the database you have links stored in.

PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Re: Spider 2.0 In reply to
That is strange, because I could connect to the database correctly using version 1.1 - running the script under the same userid. I haven`t changed anything on the Mysql-deamon.

-----------------
Wilhelm Eilertsen

Quote Reply
Re: Spider 2.0 In reply to
I still cant figure it out. The spider has been running for a long time now, at least 48 hours, an there are still no links to validate or in the database. The only thing I can find in the spider database is if I search Queue - List all, I then get a list of links, but no data and no description. Maybe I am doing something wrong, but it is hard to find out, all the documentation in the help section is for the previous version.

Quote Reply
Re: Spider 2.0 In reply to
With this error, it is because the spider is attempting to create a file called spider.status file in the admin/Plugins/Spider directory. The directory permissions must be set so that the effective userid of the spider.pl has full read/write access to that directory and file.

Hope that helps

Quote Reply
Re: Spider 2.0 In reply to
Hi Kjetil,

Could you send me telnet/links SQL login information to your site? I'd like to take a closer look.

My email address is: aki@gossamer-threads.com

For people listening in on this thread, I'll post what I find here.

Thanks

Quote Reply
Re: Spider 2.0 In reply to
I`ve chown all files under /Spider so that the webuser have access. But now I get the following error, and the install quits:

"Error running installation code: Installing Spider Hooks at (eval 256) line 228".



Sincerely,
Wilhelm Eilertsen