Gossamer Forum
Home : General : Perl Programming :

Spidering a Google query...

Quote Reply
Spidering a Google query...
 
Hi All,

I'm trying to grab a page query from google to parse, is this possible? Here's a snip of code to demonstrate my problem...

Code:
use LWP::Simple;

my $url = qq|http://www.google.co.uk/|; ## This works okay...

#my $url = qq|http://www.google.co.uk/search\?q=cartoons|; ## This doesn't!

my $file = "/_google/google.htm";

&getfile($url,$file);

exit;

sub getfile {
my ($url,$file) = @_;
print "Grab: $url\n";
my $rc = mirror($url, $file);
if ($rc == 304) {
print "Up To Date.\n";
} elsif (!is_success($rc)) {
warn "Error: $rc ", status_message($rc), " ($url)\n";
return(0);
}
}

The problem is that a 403 error is received, my guess is that I'm going to have to send a spoof header set so the request looks like IE to google... or am I wandering off in totally the wrong direction and wasting my time?


moog
-- I've spent most of my money on beer and women... the rest I just wasted.
Subject Author Views Date
Thread Spidering a Google query... moog 3563 May 21, 2003, 7:01 AM
Thread Re: [moog] Spidering a Google query...
Paul 3365 May 21, 2003, 7:09 AM
Thread Re: [Paul] Spidering a Google query...
moog 3329 May 21, 2003, 7:37 AM
Thread Re: [moog] Spidering a Google query...
Paul 3357 May 21, 2003, 7:56 AM
Post Re: [Paul] Spidering a Google query...
moog 3332 May 21, 2003, 8:28 AM