Let's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.
However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.
The "LLM giants" do both. They have to create an index to know which pages to even fetch in the first place. A simple web search does not actually necessarily hit any pages it's only when it does a page fetch based on that search
The era of robots.txt files being reasonable is gone too. Usually they either block everything, or block everything except Googlebot. If you want to make a search engine at all, you have to ignore robots.txt, ignore noindex, ignore nofollow, and so on. If we had your way, Google would have a legally enforced search engine monopoly.
This comment will be flagged because it challenges conventional wisdom.
That doesn't match my experience. For example, I recently had Meta scrape robots.txt disallowed paths. On some page they claimed they respect it but they don't. And yes, all the IPs they used to connect to me (they used a different /64 network for each connection) were owned by Meta, it was not someone pretending to be them.
