Hi Andy!
I have to wait a few hours to see if everything is right, I did this a few minutes ago ..
.htaccess now looks like:
<Files .htaccess>
order allow,deny
deny from all
</Files>
##############################
##########S P A M
##############################
order allow,deny
#####13-3-2016
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [NC]
RewriteRule .* http://google.com [F]
#####13-3-2016
deny from env=bad_bot
SetEnvIfNoCase User-Agent "^libwww" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
SetEnvIfNoCase user-agent "Indy Library" bad_bot
SetEnvIfNoCase user-agent "noxtrumbot" bad_bot
#####12-3-2016
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
deny from *.baidu.com
Deny from 180.76.
Deny from 119.63.
Deny from 123.125.
Deny from 220.181.
#####12-3-2016
allow from all
##############################
and robots,txt like:
User-agent: *
Crawl-delay: 10
Disallow:
#####12-3-2016
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
Disallow: /
##############################
BTW. Google complain because I still prevents access to the forum with .htpasswd
I must be sure that .htaccess and robots.txt works 100% before I remove .htpasswd otherwise I risk suspension again ..
One other thing - I see that Googlebot also reads pages that have part "guest=1131854"
Can it be prevented somehow?
I have to wait a few hours to see if everything is right, I did this a few minutes ago ..
.htaccess now looks like:
Code:
############################## <Files .htaccess>
order allow,deny
deny from all
</Files>
##############################
##########S P A M
##############################
order allow,deny
#####13-3-2016
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [NC]
RewriteRule .* http://google.com [F]
#####13-3-2016
deny from env=bad_bot
SetEnvIfNoCase User-Agent "^libwww" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
SetEnvIfNoCase user-agent "Indy Library" bad_bot
SetEnvIfNoCase user-agent "noxtrumbot" bad_bot
#####12-3-2016
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
deny from *.baidu.com
Deny from 180.76.
Deny from 119.63.
Deny from 123.125.
Deny from 220.181.
#####12-3-2016
allow from all
##############################
and robots,txt like:
Code:
############################## User-agent: *
Crawl-delay: 10
Disallow:
#####12-3-2016
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
Disallow: /
##############################
BTW. Google complain because I still prevents access to the forum with .htpasswd
I must be sure that .htaccess and robots.txt works 100% before I remove .htpasswd otherwise I risk suspension again ..
One other thing - I see that Googlebot also reads pages that have part "guest=1131854"
Can it be prevented somehow?
