Gossamer Forum
Quote Reply
Andy's Spider Bug??
When I spider some sites, I get the "Restricted!" in the title...
The pages that this comes up on the html of the page is like:
<title>
This is the title.
</title>

if the title is like <title>This is the title.</title>, it spiders okay.

I went into the Spider.pm and changed the
foreach (@html) {
m,<title>(.+?)</title>,i and $title = $1;
}

to

foreach (@html) {
m,<title>
(.+?)
</title>,i and $title = $1;
}

and it spiders the questionable pages okay, but then the ones that were normal from the beginning say Restricted!...
Any solution so it picks up either...

</not a clue>
Subject Author Views Date
Thread Andy's Spider Bug?? Dinky 3738 Apr 29, 2004, 7:21 PM
Thread Re: [Dinky] Andy's Spider Bug??
Andy 3638 Apr 30, 2004, 1:07 AM
Thread Re: [Andy] Andy's Spider Bug??
pugdog 3644 Apr 30, 2004, 4:19 AM
Thread Re: [pugdog] Andy's Spider Bug??
Andy 3624 Apr 30, 2004, 5:41 AM
Post Re: [Andy] Andy's Spider Bug??
Dinky 3622 Apr 30, 2004, 5:49 AM
Thread Re: [Dinky] Andy's Spider Bug??
katabd 3637 Apr 30, 2004, 7:23 AM
Thread Re: [katabd] Andy's Spider Bug??
Andy 3630 Apr 30, 2004, 8:01 AM
Thread Re: [Andy] Andy's Spider Bug??
katabd 3619 Apr 30, 2004, 8:33 AM
Thread Re: [katabd] Andy's Spider Bug??
Dinky 3599 Apr 30, 2004, 7:11 PM
Thread Re: [Dinky] Andy's Spider Bug??
Andy 3597 May 1, 2004, 1:29 AM
Thread Re: [Andy] Andy's Spider Bug??
Dinky 3542 Jun 26, 2004, 6:37 PM
Post Re: [Dinky] Andy's Spider Bug??
Andy 3528 Jun 27, 2004, 3:48 AM