Hi there.
When setting up a robots.txt file what are the advantages and disadvantages of disallowing a cgi-bin?
It seems like all the sample robots.txt files I've found disallow the cgi-bin directory, but a lot of my content is dynamically generated, and it would be nice if the search spiders could index it. Is it that the robots aren't able to read the cgi generated content properly? Do they try to read the .cgi or .pl files themselves? Does it tax the server too much? I'm just not sure I underestand enough about how these robots work to know what I should be expressly disallowing.
Thanks in advance for any advice.
Best,
Adam
Fractured Atlas :: Liberate the Artist
Services: Healthcare, Fiscal Sponsorship, Marketing, Education, The Emerging Artists Fund
When setting up a robots.txt file what are the advantages and disadvantages of disallowing a cgi-bin?
It seems like all the sample robots.txt files I've found disallow the cgi-bin directory, but a lot of my content is dynamically generated, and it would be nice if the search spiders could index it. Is it that the robots aren't able to read the cgi generated content properly? Do they try to read the .cgi or .pl files themselves? Does it tax the server too much? I'm just not sure I underestand enough about how these robots work to know what I should be expressly disallowing.
Thanks in advance for any advice.
Best,
Adam
Fractured Atlas :: Liberate the Artist
Services: Healthcare, Fiscal Sponsorship, Marketing, Education, The Emerging Artists Fund