In server logs you can distinguish the type of LLM's traffic. OAI-SearchBot/GPTBot/ClaudeBot is just indexing but ChatGPT-User/Claude-User etc. may be valuable traffic generated by user's conversation with LLM's. So it's worth choosing which ones you block, instead of blocking all of them.
What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.
Tons of people click from ChatGPT! It's now the top referrer on most of my websites, and seems to be on path to overcoming Google soon (a ton of traffic from Google has vanished since the AI Overview rollout...).
I realized its much faster to do the "google thing" just with chatgpt. Its filters all the sh!t for you. Searching the net frustrates me most of the times this days. Search engines are software for an older generation of the www.
In my case this is the main source of registered users. If a user is engaged in a discussion with chatbot and gets the recommendation from it his engagement is higher. For example, in my system, a user has registered and within one hour used the system 7 times and stayed active while for registrations from Google I can observe zero-activity accounts.
Strong agree, a good way forward is letting user agent once again truly mean an agent on behalf of the user. And speaking anecdotally, I've seen different traffic "shapes" under the same UA, which leads me to suspect that some actors might be disguising traffic as more mainstream labs.
