Gossamer Forum: Products: Gossamer Links: Development, Plugins and Globals: Spider plug in problem

katabd · Jul 26, 2003, 3:52 AM

Hi

When spidering any list of sites using Andy's spider plug in I am getting the following error after spidering few links only..

Content-type: text/html

Software error:

Invalid argument at /home/virtual/site315/fst/var/www/cgi-bin/links/admin/Net/Whois/Raw.pm line 132.

Any ideas?
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

Andy · Jul 26, 2003, 4:18 AM

Could you give me an example URL for when this happens?

Cheers

Andy (mod)
andy@ultranerds.co.uk

katabd · Jul 26, 2003, 4:35 AM

Hi

The following USL

http://abujasir.tripod.com/...-centers-mosques.htm

Spider stops at:

URL: http://icc-chicago.org/ Title: Access Disabled Description: Author (owner): Webmaster Email (owner): not@available.org Options: Status: WARNING: Link already exists!

Content-type: text/html Software error:

Connection refused at /home/virtual/site315/fst/var/www/cgi-bin/links/admin/Net/Whois/Raw.pm line 132.
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

Andy · Jul 26, 2003, 6:09 AM

Weird....it works ok for me. Can you try adding the following in admin/Plugins/Installed/Spider.pm ;

~ line 275, just after the beginning of the 'foreach';

Code:
next if $_ =~ /^\s/;

Does that fix it?

Cheers

Andy (mod)
andy@ultranerds.co.uk

katabd · Jul 26, 2003, 10:55 PM

Hi Andy

That did solve the issue..

I have two suggesttions:

1- The category selection option that we talked about earlier.

2- The ability to grab the real URL not the redirection on.. example page.cgi?ID=2154 versus http://domain.com/page.html

Thanks as always.
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

Andy · Jul 28, 2003, 1:14 AM

Glad to hear it. I'll make that fix in the new version.

1) Its been pretty busy here recently, could you please run it by me again? Crazy

2) The problem with this, is that I'm not really sure how to get the final URL. If anyone knows how I could use LWP::Simple to acomplish this (i.e not grab the URL, but instead the final page that the user would be sent to), that would be great.

Cheers

Andy (mod)
andy@ultranerds.co.uk

katabd · Jul 28, 2003, 3:31 AM

In Reply To:

1) Its been pretty busy here recently, could you please run it by me again? Crazy

Well in the first page if we can give the admin the option to choose a category to import all the links to (through select or entering a specific category ID number). That will be most helkpfull when importing a list of URL's to a specific category so we will not have to select it again for each link.
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

mgeyman · Jul 28, 2003, 8:14 AM

Andy,

Sounds like some great additional helpful features. It would be great to streamline this process as much as possible!

Also, is it possible for the spider to identify other pages, for instance the site's Contact Page, for the site's email address if it isn't located on the site's home page?

Thanks. Keep up the great work!

mgeyman

Andy · Jul 28, 2003, 8:20 AM

I'm *very* busy at the moment with custom work, but once that is done, I'm gonna hopefully make the category addition mod to this Plugin, and also a couple more (hopefully). I'm also gonna be finishing off my 'link of the week' plugin, which was almost completed when I got this big influx of work.

I'll keep you all posted.

Cheers

Andy (mod)
andy@ultranerds.co.uk

Andy · Aug 1, 2003, 4:08 PM

Ok, I've added the category choice option at the beginning of the process. You can either choose a global category, or not define one at all.

The online demo has been updated too. Please find details of the demo here.

Existing members can grab their update from http://new.linkssql.net/page.php?page=account .

For more details on this plugin, please see;

http://new.linkssql.net/...=fee&page=Spider

Cheers

Andy (mod)
andy@ultranerds.co.uk

katabd · Aug 3, 2003, 4:09 AM

Thanks Andy

Looks great..

What does the Restricted! means?

I am getting it on lots of the active links..
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

katabd · Aug 3, 2003, 6:12 PM

Hi Andy

Did you fix this in the new release, because I am getting the same error again.
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

Andy · Aug 4, 2003, 1:46 AM

>>>Did you fix this in the new release, because I am getting the same error again.
<<<

Fix what? The 'Restricted' thing? That is simply there to let you know that the spidered page has been blocked by robots.txt. You can disable the use of robots.txt in the Plugins settings.

Cheers

Andy (mod)
andy@ultranerds.co.uk

katabd · Aug 4, 2003, 3:12 AM

Sorry Andy

I was talking about

Software error:

Connection refused at /home/virtual/site315/fst/var/www/cgi-bin/links/admin/Net/Whois/Raw.pm line 132.
Regards
KaTaBd

Users plug In - Multi Search And Remote Search plug in - WebRing plug in - Muslims Directory

Andy · Aug 4, 2003, 3:26 AM

Hi. Sorry about that. I'm not sure why that bug fix wasn't made in this version. Please try the latest one, available from your 'Members Area'.

Cheers

Andy (mod)
andy@ultranerds.co.uk

Spider plug in problem

Last edited by: