Gossamer Forum: General: Internet Technologies: WebStripper

Feb 7, 2003, 12:58 AM

yogi

Veteran (2199 posts)

Feb 7, 2003, 12:58 AM

Post #1 of 9

Shortcut

WebStripper

Has anyone got experience with WebStripper?

I just noticed in my apache logs, that somebody using webstripper has downloaded my whole site, including all dynamic pages on the forum. Shouldn't a well behaved robot know that spidering dynamic pages does not make much sense?

Ivan
-----
Iyengar Yoga Resources / GT Plugins

Feb 7, 2003, 1:18 AM

Andy

Veteran / Moderator (17364 posts)

Feb 7, 2003, 1:18 AM

Post #2 of 9

Shortcut

Re: [yogi] WebStripper In reply to

Site strippers like that should be banned. It takes the whole fun out of making a distinct site, that no-one else has. I mean, whats the point in having a 100% copied site? Unimpressed

Regarding stopping them. I'm not sure where...but I'm sure Paul came up with some code to put in Apache that would stop them being allowed to spider your server/sites Smile

Andy (mod)
andy@ultranerds.co.uk

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

Feb 7, 2003, 2:09 AM

Paul

Veteran (19537 posts)

Feb 7, 2003, 2:09 AM

Post #3 of 9

Shortcut

Re: [yogi] WebStripper In reply to

Code:
RewriteEngine On  
RewriteCond %{HTTP_USER_AGENT} ^.*EmailSiphon     [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*Wget            [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*EmailWolf       [NC,OR]  
RewriteCond %{HTTP_USER_AGENT} ^.*HTTrack         [NC] 
RewriteRule ^.*$               http://www.aol.com [L]

Just keep adding naughty bots to the list - don't use [OR] for the last one in the list.

It's too late now your site has been grabbed but you can at least prevent anything more.

Last edited by:

Paul: Feb 7, 2003, 2:10 AM

Feb 7, 2003, 5:44 AM

DogTags

Veteran (1345 posts)

Feb 7, 2003, 5:44 AM

Post #4 of 9

Shortcut

Re: [Paul] WebStripper In reply to

What does the last line do? Does this bounce the stripper to whatever url is shown below?

Quote:

RewriteRule ^.*$ http://www.aol.com [L]

Thanks

------------------------------------------
military dog tags for personal identification, travel, and many other uses

Last edited by:

DogTags: Feb 7, 2003, 5:50 AM

Feb 7, 2003, 6:17 AM

Paul

Veteran (19537 posts)

Feb 7, 2003, 6:17 AM

Post #5 of 9

Shortcut

Re: [DogTags] WebStripper In reply to

Yeah - if it tries to grab your site it will just grab aol.com a load of times ;)

Feb 7, 2003, 6:24 AM

DogTags

Veteran (1345 posts)

Feb 7, 2003, 6:24 AM

Post #6 of 9

Shortcut

Re: [Paul] WebStripper In reply to

Ya know, I should have asked the following, too:

Is the bounce line necessary, or can you just have the stripper lines?

I think I'd prefer preventing the strippers, but not bouncing to anywhere.

Thanks Smile

------------------------------------------
military dog tags for personal identification, travel, and many other uses

Feb 7, 2003, 6:48 AM

Thomas.

Veteran (1210 posts)