When I spider some sites, I get the "Restricted!" in the title...
The pages that this comes up on the html of the page is like:
<title>
This is the title.
</title>
if the title is like <title>This is the title.</title>, it spiders okay.
I went into the Spider.pm and changed the
foreach (@html) {
m,<title>(.+?)</title>,i and $title = $1;
}
to
foreach (@html) {
m,<title>
(.+?)
</title>,i and $title = $1;
}
and it spiders the questionable pages okay, but then the ones that were normal from the beginning say Restricted!...
Any solution so it picks up either...
</not a clue>
The pages that this comes up on the html of the page is like:
<title>
This is the title.
</title>
if the title is like <title>This is the title.</title>, it spiders okay.
I went into the Spider.pm and changed the
foreach (@html) {
m,<title>(.+?)</title>,i and $title = $1;
}
to
foreach (@html) {
m,<title>
(.+?)
</title>,i and $title = $1;
}
and it spiders the questionable pages okay, but then the ones that were normal from the beginning say Restricted!...
Any solution so it picks up either...
</not a clue>