Innocent until proven guilty. Every citizen has rights until there is reasonable suspicion, probable cause, etc that they are commiting a crime.
The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated...
Is this a point you're arguing seriously? If you find someone's plans to, for example, shoot up a school crumpled in a note on the ground that should be...ignored? Or what? I struggle to comprehend what you're actually arguing for here.
I think I pretty much agree with you that if someone finds evidence of danger to someone else's life then it is a fair expectation that they do something with that evidence. And despite that I still believe that is not a good enough reason to eliminate someone's right to privacy, and yet still that right doesn't mean that we should be able to demand privacy in all products. I think it is fairly clear that these big LLMs are not private products and we shouldn't expect them to be, but I'm still in support of offering private alternatives that people could use without having every input scrutinized.