Hi,

i have a problem with google.

if have a robot.txt file with disallow: /cgi-bin/ but google indexes all my

http://www.domain.de/...s/jump.cgi?ID=number

links, naturally without an description. google indexes also the deadlinks.cgi and so on.

Is there any way to stop google from indexing cgi links.

I use also sitemaps, but the cgi links are still in the google index.

Best,
Karl

Your robots.txt should look like this:

User-Agent: *
Allow: /
Disallow: /cgi-bin

Straight from the horses' mouth (Google): http://www.google.com/webmasters/remove.html#exclude_pages
http://www.google.com/webmasters/bot.html#robots

See this page for general robots.txt info: http://www.webmasterstop.com/59.html


Leonard
aka PerlFlunkie

Hi Leonard,

thank you!

My robots.txt looks like this, but with a trailing slash behind the cgi line. Never saw it without the slash. I will give it a try.

Best,
Karl