... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".

Which is why some search engines simply ended up following only the directives given to googlebot, and ignoring the rest.

> All robots forbidden, except googlebot

After which people keep claiming google search is better, not just given special treatment.