We shouldn't use an LLM because...LLMs never improve?
This seems like a great source for domain-specific RL, but even without that, we can expect there to be higher-quality LLMs that can be trivially swapped in within a year, if not a week.
Okay, but I don't understand what you're getting at with them not learning? They acquire more information, get better at providing accurate and relevant information, and acquire new skills. What is the learning they're not doing?
Maybe a human should be available as a backup or something, but not because LLMs don't "learn" -- the system seems to learn in all relevant senses.
(I'd also object to the no accountability, but another subthread is already on that.)
This seems like a great source for domain-specific RL, but even without that, we can expect there to be higher-quality LLMs that can be trivially swapped in within a year, if not a week.