There is a kind of proof from lots of users getting apparent value out of it, sure. But we also have automated evals for AI systems, it's a well developed technique at this point, and it would bootstrap that process by providing some evidence.

I think AI/ML systems are unique compared to other software in this regard, because the "how it works" is not inherent in the software itself, and not working once doesn't mean that it doesn't work at all, it could have been luck.