Search engines that obey robots.txt are stealing?

If someone does not want on a search engine index and they say not to index, and then get indexed anyhow is stealing.

But a “please read my content and make available to your users” then claim doing so is stealing seems a bit out there.

What am I missing?

There is a difference between "read my content and make it available to your users" and "make money by reading my content and making it available to your users". Terms of use like this are trying to say "we are the only ones who get to make money from this", and therein lies the problem.

To sharpen the distinction slightly (about increasingly abusive search-engines):

1. "You may make my content findable to your users as a service to them, taking a finder's-fee from ads etc."

2. "You may plagiarize my content, serving that to your users and cutting me out of the user-traffic almost entirely."

> we are the only ones who get to make money from this

Which, somewhat ironically, is the stance of copyright maximalists decrying “theft” be aggregators and AI while paying not a cent to the people whose shoulders they stand on.

It seems the complaint isn’t so much about trying to claim exclusive perpetual rights to an incremental creative layer built on millennia of history, but about the wrong people making the claim.

Well said. I'm sure a subset of the content that Cloudflare has indexed also has terms like Cloudflares, that they happily ignored.