If they only collect the data for analysis I guess this is fine (they already get way more sensitive data from users anyways, so if privacy is your concern you've made the mistake many steps ago). The much more interesting question is if they directly act on this data in their API. For example by rate-limiting, compute-limiting or rerouting to weaker models. That might even be legally questionable. I would really like to see this as a follow-up analysis, but I guess it is way more difficult and will also cost quite a bit in tokens.
I've heard that it was possible to trigger really obvious output poisoning on Fable with something as basic as asking the model to think outside of its built-in hidden thinking delimiters.
I'm thinking more of EULAs. Even if Anthropic somehow wedges this into their TOS, it might still be illegal. For example, in many US states this could potentially be classified as consumer fraud. You can't just sell one thing and then secretly and intransparently turn it into something else before shipping it. And in the EU it might violate GDPR too.
"If they only collect the data for analysis I guess this is fine"
I think you missed the memo on how foolish this attitude is. It came out around the time Edward Snowden made his discoveries at the NSA public. I suggest you look into it
This is not some fundamental problem. There are amazing options that don't suffer this downside. Most people are just too lazy and want something that works out of the box, so they accept or ignore the inherent risks of tools like Claude Code or Codex. But you just have to invest a little bit of research into what you need and which open model and harness suits your requirements. Then you can have your cake and eat it too. Long term this will be the standard anyways, it's only difficult while the frontier is moving so fast and aimed at ridiculous IPOs.
