Hi.
gogglebot was hammering server like mad often triggering mod_dosevasive and it was on forum that it was happening most aggressively. Contacted 'G' and this is the reply:
Hi <Edited>,
Thank you for your reply. As you've observed, session IDs can cause
problems for our robots. While we can reduce the crawl rate for this page,
you can disable session IDs for Googlebot so that our robots may crawl
your site more efficiently and not affect your crawl rate.
Regards,
The Google Team
Is it possible to eliminate guest=xxxxxx from the url? Apparently each guest=xxxxxx for the same url (post/forum/etc etc) is being viewed as a different url by bot and hence the hammering. MSN hammering is also of a similar scale, the diference being that MSN doesn't care to bother. (Even their IE 6 causes hammering in certain cases, perhaps a ploy to jack up their "Browser Stats As Analyzed by Log Analyzers")
Thanks
HyTC
gogglebot was hammering server like mad often triggering mod_dosevasive and it was on forum that it was happening most aggressively. Contacted 'G' and this is the reply:
Hi <Edited>,
Thank you for your reply. As you've observed, session IDs can cause
problems for our robots. While we can reduce the crawl rate for this page,
you can disable session IDs for Googlebot so that our robots may crawl
your site more efficiently and not affect your crawl rate.
Regards,
The Google Team
Is it possible to eliminate guest=xxxxxx from the url? Apparently each guest=xxxxxx for the same url (post/forum/etc etc) is being viewed as a different url by bot and hence the hammering. MSN hammering is also of a similar scale, the diference being that MSN doesn't care to bother. (Even their IE 6 causes hammering in certain cases, perhaps a ploy to jack up their "Browser Stats As Analyzed by Log Analyzers")
Thanks
HyTC