> weights update over time and past learning improves future learning such that we get better sample efficiency

Is there an architecture-independent definition of forward transfer?

For the practical experience and implications of AI progress, I think we are increasingly discussing what these LLMs can accomplish inside a stateful harness, the state of which could be described as part of a (very squirrely) parameter space.