It interleaves random streams of 32K characters long each when reading the whole corpus, but each such stream reads continuously as you would expect. This is a necessary step to prevent just normal, not catastrophic, forgetting. I have not tested it in any other regimes yet with bigger or smaller windows. You can imagine a person that changes the activity from time to time, so I think it is justified. So there is not really "early in the stream".
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.
I have my own solution here that also works for sharing stuff with clients with configurable whitelists at artifacts.iofold.com (also open-source for self hosting on cf workers for almost free, github linked on the homepage)
I'm working on https://artifacts.iofold.com, a way to use artifacts (self contained html + optional assets like image/video/json) easily across agents, with feedback loop and docsend style wttribiti gates.
Have made it agent friendly enough that my teammates' agents can read and drop commennts on specs/storyboards etc, and my agent can close the loop by iterating with a new artifact version.
I'm very curious to see a benchmark for this - have toyed with the idea myself but haven't put in the hard work to test these hypothesis on extracting learning signal from deep-agent traces.
The profile as a memory primitive is pretty interesting, how do you scale it for a scoped access/hierarchical setting? (for example, overall memory of processes => customer specific => project specific memory)
I've struggled with adding evals to my AI agents for last few months, and felt that vibe evals should have a path to building a robust system down the line.
Working on a plugin for langfuse to create evals functions and dataset from ingested traces automatically, based on ad-hoc user feedback.
