Gossamer Forum
Home : Products : Gossamer Forum : Discussion :

Googlebot and guest=xxxxxxxxx

Quote Reply
Googlebot and guest=xxxxxxxxx

gogglebot was hammering server like mad often triggering mod_dosevasive and it was on forum that it was happening most aggressively. Contacted 'G' and this is the reply:

Hi <Edited>,

Thank you for your reply. As you've observed, session IDs can cause
problems for our robots. While we can reduce the crawl rate for this page,
you can disable session IDs for Googlebot so that our robots may crawl
your site more efficiently and not affect your crawl rate.

The Google Team

Is it possible to eliminate guest=xxxxxx from the url? Apparently each guest=xxxxxx for the same url (post/forum/etc etc) is being viewed as a different url by bot and hence the hammering. MSN hammering is also of a similar scale, the diference being that MSN doesn't care to bother. (Even their IE 6 causes hammering in certain cases, perhaps a ploy to jack up their "Browser Stats As Analyzed by Log Analyzers")


Last edited by:

HyperTherm: Mar 4, 2005, 3:21 PM