Gossamer Forum: General: Internet Technologies: WebStripper

Feb 7, 2003, 12:58 AM

yogi

Veteran (2199 posts)

Feb 7, 2003, 12:58 AM

Post #1 of 9

Shortcut

WebStripper

Has anyone got experience with WebStripper?

I just noticed in my apache logs, that somebody using webstripper has downloaded my whole site, including all dynamic pages on the forum. Shouldn't a well behaved robot know that spidering dynamic pages does not make much sense?

Ivan
-----
Iyengar Yoga Resources / GT Plugins

Feb 7, 2003, 1:18 AM

Andy

Veteran / Moderator (18441 posts)

Feb 7, 2003, 1:18 AM

Post #2 of 9

Shortcut

Re: [yogi] WebStripper In reply to

Site strippers like that should be banned. It takes the whole fun out of making a distinct site, that no-one else has. I mean, whats the point in having a 100% copied site? Unimpressed

Regarding stopping them. I'm not sure where...but I'm sure Paul came up with some code to put in Apache that would stop them being allowed to spider your server/sites Smile

Andy (mod)
andy@ultranerds.co.uk

Feb 7, 2003, 2:09 AM

Paul

Veteran (19537 posts)

Feb 7, 2003, 2:09 AM

Post #3 of 9

Shortcut

Re: [yogi] WebStripper In reply to

Code:
RewriteEngine On  
RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon     [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*Wget            [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf       [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack         [NC] 
RewriteRule ^.*$               http://www.aol.com [L]

Just keep adding naughty bots to the list - don't use [OR] for the last one in the list.

It's too late now your site has been grabbed but you can at least prevent anything more.

Last edited by:

Paul: Feb 7, 2003, 2:10 AM

Feb 7, 2003, 5:44 AM

DogTags

Veteran (1345 posts)

Feb 7, 2003, 5:44 AM

Post #4 of 9

Shortcut

Re: [Paul] WebStripper In reply to

What does the last line do? Does this bounce the stripper to whatever url is shown below?

Quote:

RewriteRule ^.*$ http://www.aol.com [L]

Thanks

------------------------------------------

Last edited by:

DogTags: Feb 7, 2003, 5:50 AM

Feb 7, 2003, 6:17 AM

Paul

Veteran (19537 posts)

Feb 7, 2003, 6:17 AM

Post #5 of 9

Shortcut

Re: [DogTags] WebStripper In reply to

Yeah - if it tries to grab your site it will just grab aol.com a load of times ;)

Feb 7, 2003, 6:24 AM

DogTags

Veteran (1345 posts)

Feb 7, 2003, 6:24 AM

Post #6 of 9

Shortcut

Re: [Paul] WebStripper In reply to

Ya know, I should have asked the following, too:

Is the bounce line necessary, or can you just have the stripper lines?

I think I'd prefer preventing the strippers, but not bouncing to anywhere.

Thanks Smile

------------------------------------------

Feb 7, 2003, 6:48 AM

Thomas.

Veteran (1220 posts)