Because search engines were built because they can crawl sites and if there is a monopoly, adhering to robots.txt just support them.
It is important to note that I am not saying that there should be indiscriminate crawling that breaks your site, just saying that robots.txt is not the solution.
Well that I agree with, though mainly because robots.txt is only an ask and can be ignored by scrapers.
I do prefer site owners being able to limit how their content can be used though. That's individual choice much like choosing a software license for open source code, and totally unrelated to net neutrality as far as I understand it.
It is important to note that I am not saying that there should be indiscriminate crawling that breaks your site, just saying that robots.txt is not the solution.