I used mod_rewrite to make my links content search engine friendly, and since I've imported a major ODP category for data, there's quite a bit there for old Googlebot to feast on. As is its practice during the first week of the month, Googlebot has been indexing my site, and for the most part, it appears to be handling all those links pages just fine - requesting between 2-5 pages per second, around the clock.
One interesting thing I've noticed, though, is that there have been a handful of times where 10-15 consecutive requests would all result in 500 internal server errors. When I manually checked the urls in question, they worked just fine. I'm thinking these errors must have had something to do with Googlebot's rapid devouring of what appeared to be static html pages, but which were in fact being generated by page.cgi requests. Is there any reason, though, why page.cgi might become "overloaded" somehow and respond with 500 internal server errors? I'm thinking this is probably nothing to worry about, but I'd appreciate an expert opinion.
Thanks very much.
Fractured Atlas :: Liberate the Artist
Services: Healthcare, Fiscal Sponsorship, Marketing, Education, The Emerging Artists Fund
One interesting thing I've noticed, though, is that there have been a handful of times where 10-15 consecutive requests would all result in 500 internal server errors. When I manually checked the urls in question, they worked just fine. I'm thinking these errors must have had something to do with Googlebot's rapid devouring of what appeared to be static html pages, but which were in fact being generated by page.cgi requests. Is there any reason, though, why page.cgi might become "overloaded" somehow and respond with 500 internal server errors? I'm thinking this is probably nothing to worry about, but I'd appreciate an expert opinion.
Thanks very much.
Fractured Atlas :: Liberate the Artist
Services: Healthcare, Fiscal Sponsorship, Marketing, Education, The Emerging Artists Fund