I have a dedicated server set up with much of the dmoz imported, its running in dynamic but with a mod rewrite to make the links seem static, the spiders are eating it up nicely, with one problem, the spiders are accessing all the other links too, all the cgi being run was putting a load on my old server which was bringing it down, yes, spidering alone was overloading it, who would think that was a thing to complain about :)

Upgrade to P4 2.8 1mb ht cache and dual 160 sata raid drives seems to have given it enough power to cope, nevertheless I would like to minimize the demand on resources.

I have put in the robots.txt to ignore the cgi-bin and all the cgi's in question such as user.cgi and review.cgi but they continue to be spidered!

What I would like is for the directory pages and the links to be spidered but nothing else, anyone know a way?

I read google recently has a facility to look for a nofollow tag in the link but other than that, I've found nothing to give me a clue what to try next.

Anyone able to suggest something?