Nope. Even if context can theoretically encode arbitrary computation under fixed weights, this requires the weights to implement a usable interpreter. Random weights almost surely do not. Training is what constructs that interpreter. Without it context has no meaningful computational semantics.
It's kind of like asking if I make a random circuit with logic gates, does that become a universal computer that can run programs.
There has been a lot of talk about how continual learning might be "just and engineering challenge" and that we could have agents that continuously learn from experience by just having longer and longer context windows.
What I am trying to argue for in the article is how such a view might be misplaced - just extending the context length and adding more instructions in the context will not get you continual learning - the representational capacity of weights will be the limiting factor.
Just a fun way to think about it. Would love to hear your thoughts.
>just extending the context length and adding more instructions in the context will not get you continual learning...
I agree. But I am wondering if context would help in answering superficial questions and only fail when answering questions that require deeper understanding.
I'd say the way to think about it is in terms of the questions you ask being in-distribution or out of distribution w.r.t the model training dataset.
Consider this, if something fundamental has changed in the world after the model was released(ie after the knowledge cut off date), then it would be very difficult for the model to reason about it. One concrete example is the the following: If you ask Opus or any decent coding model to do effort estimation on a coding task, then it would come up with multi week timelines - the models themselves doesn't know that because "they exist", these timelines have now been slashed to a few hours - you can try saying this in the prompt, however, they don't seem to internalise this.
Imagine an LLM that can also OCR. Would it be possible to make it OCR a totally new letter by only showing a single picture of it and including the fact in the context?
I think it would not be possible. That would be a good demonstration of the point I (and possibly you as well) is trying to get across.
I spent the last weekend thinking about continual learning. A lot of people think that we can solve long term memory and learning in LLMs by simply extending the context length to infinity. I analyse a different perspective that challenges this assumption.
Also interesting to think about: could a single system be generally intelligent, or is a certain bias actually a power. Can we have billions of models, each with their own "experience"
I think both the views have their merits. In my mind the hardware vs software analogy for weights vs context holds better because in most modern computing systems, the hardware is fixed and the software changes. What the system can do efficiently, in practice, is a function of both the limitations/capabilities of the hardware and the software their respective capability ceilings.
The brain theory also kind of says the same thing, but it's hard to say what stays fixed vs changes with experience in the brain ig.
Another way I see it is... Mind is process. LLM is (very lossy) snapshotted state of process/mind. LLM in-process is mind-emulator with potential to explore the state-space of the mind-snapshot. Consequently, and by its very construction, LLM cannot be mind.
We ran this experiment a few weeks ago, but Anthropic’s Mythos report dropped this week and thought this would be relevant to share now.
Surprisingly, we found that for repos already in the training data(pre cut off set), the contamination is at such a high level that even by just giving the model the file name and not file contents, the model is able to tell what is inside that file. Same for file paths. Just given a file name, the model is able to correctly guess the file path - this implies that models already know the structure of these repos and thus understand what to look for and where.
This ability drops sharply for unseen repos(post cutoff set) - raises the question how effectively it will hold for private repos with proprietary scaffoldings and programming patterns. Then the question worth asking is how much of Mythos's capability on well known codebases like Firefox and OpenBSD is genuine reasoning vs parametric familiarity with their structure?
Methodology caveat: modest sample (9-10 repos per group), treat numbers as directional - more experiments in progress....
The 1-2 files per request constraint is interesting. I do the same
thing, but I've started thinking of it as a symptom rather than a
solution. The reason you have to constrain scope is because the
agent doesn't know what it doesn't know — it can't tell you "this
change will break something 4 layers away." So you manually limit
blast radius as a substitute for actual understanding.
Doesn't feel like that's where this should end up?
Personal superintelligence sounds nice until you actually try to use it.
We spent time yesterday arguing through an architecture decision. Today I ask the Agent to help implement it - it knows nothing about any of that. You’re effectively starting over.
Feels like the real problem isn’t intelligence, it’s continuity. And most benchmarks don’t even touch that.
Yes this feels very new from a product and harness design perspective but it's brand new! Nine months old. The mobile and web sessions don't even real-time sync between each-other yet there's endless work to be done and time will tell if they can bring all the people to bring it together. The underlying model seems like a great foundation now but securing the supremacy of usage is multilateral requiring both machine learng advancements and product/harness/usage design.
Solving continual learning is key to AGI - and we maybe close to solving it.