Gossamer Forum: General: Perl Programming: Grab actual URL...

Oct 9, 2002, 1:17 AM

Andy

Veteran / Moderator (18441 posts)

Oct 9, 2002, 1:17 AM

Post #1 of 6

Shortcut

Grab actual URL...

GT's scripts have been causing problems again with grabbing other sites URL's Tongue

I'm writing a spider plugin for LSQL, and use the following to grab the HTML from the page;

my @html = get($_);

The problem I am having, is that some of the spidered pages (to grab other new links) are refering to jump.cgi, on their domain. What I really need to grab is the final URL that the spider is sent to....anyone got any ideas? I'm stumped Frown

Cheers

Andy (mod)
andy@ultranerds.co.uk

Oct 9, 2002, 2:56 AM

Paul

Veteran (19537 posts)

Oct 9, 2002, 2:56 AM

Post #2 of 6

Shortcut

Re: [Andy] Grab actual URL... In reply to

...you mean you want to follow the hyperlinks?

That's kind of the beauty of spiders....they are written to do that Wink

You need to grab all links, then fetch them then parse etc. up to a certain number of levels deep probably specified as an admin option.

You could do with using parrallell fetching too.

Oct 9, 2002, 3:32 AM

Andy

Veteran / Moderator (18441 posts)

Oct 9, 2002, 3:32 AM

Post #3 of 6

Shortcut

Re: [Paul] Grab actual URL... In reply to

Yes, I know thats what spiders are meant to do, which is why I'm trying to do it Wink

I still don't understand how you are saying I could grab the actual URL, rather than jump.cgi :-/

Cheers

Andy (mod)
andy@ultranerds.co.uk

Oct 9, 2002, 3:53 AM

Paul

Veteran (19537 posts)

Oct 9, 2002, 3:53 AM

Post #4 of 6

Shortcut

Re: [Andy] Grab actual URL... In reply to

I don't get why you need to grab the real URL. If you spider a page and jump.cgi happens to be one of the hyperlinks, then so be it. It will still point to the right place.

Oct 9, 2002, 4:43 AM

Andy

Veteran / Moderator (18441 posts)

Oct 9, 2002, 4:43 AM

Post #5 of 6

Shortcut

Re: [Paul] Grab actual URL... In reply to

Yes,but it needs to be stored in the LSQL database as the actual URL, and not the jump.cgi one Wink

Andy (mod)
andy@ultranerds.co.uk