What's the advantage of doing this, versus becoming good at context management and RAG? I always found trained knowledge unreliable, given that it is lossy by construction.
That's a total non sequitur. Training an LLM creates a reasoning engine. The residual knowledge from that does not compare to how humans learn knowledge.
Can you answer your original question now? If not, you should think about the similarities between humans and LLMs more rather than get lost in technical pedantry.
Or think about an extreme case: an LLM that is trained on almost nothing combined with great RAG and context.
I asked my question because I have an opinion, and I wanted to hear the creator's perspective. Your question had nothing to do with any of it, and is irrelevant.
You can call it "technical pedantry", but what you actually meant is "precision and logic". Which is relevant to make a connection. Your new fallacy is called "ad hominem", btw.
And there is one more logical fallacy hidden in your message: training an LLM on a lot of data is not the same thing as then relying on that lossy data once training is done.
I always found the frontier AI labs' focus on trained knowledge a bit confusing. From my perspective, training creates a reasoning engine. Beyond that, models reason about data from more reliable sources. I wonder why this is not a distinction they make?
