In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.
In principle I’d agree usually, but I think this specific case, it’s reasonable to highlight that the “experts get value” argument doesn’t really add much to the discourse anymore.
We know this and you just need to look at what’s happening with LLMs in maths to see it’s true. It doesn’t add anything to the verification point however.
While this may be a common understanding in this community, it's worth repeating so that this community can tell everyone else, as I don't think most people understand this currently.
But also, it is a post that is self promoting their personal twitter page; there should be a high threshold of doing this, and regurgitating some trite analogy should fall below the bar.
I've had an idea for an automated verification for a while, but too busy to implement it.
For every property of the system, implement a mock that simulates a system that works, and multiple variations with forseeable bugs (models are fairly good at predicting possible bugs). Then, write a unit tests that pass on the working simulation and fail on the buggy simulations.
This way model ensures reasonably well that the unit tests test what they're supposed to. Then, you use those tests as a verification to guide the model to implement the functionality.
This approach relies on the assumption that the model can design a reasonable architecture composed of independently-testable modules. In my experience, frontier models should be able to do that.
I've been enforcing TDD with Claude since the start and to be honest it actually works great. It doesn't write bad tests just to fit a bad implementation. And when running against a Pglite in-memory database it can iterate rapidly.
I think self-verification can work in some ways. For example, I (well, an LLM) recently wrote some caching optimizations for a particular filesystem intensive workload on a cluster. This is an internal thing, not customer facing, for context.
Deploy to the staging cluster, have the LLM monitor with read access to the logs. Opus is… better than a human at looking at logs and comparing timings and checking for warnings and errors.
If it passes a small load test there, deploy to the live cluster and actively monitor. If it doesn’t fall over under load, is faster, is still correct, not throwing errors it works good enough.
Does much more matter? This produced a fairly large performance improvement. Verification in this case is the process of running it under load and double checking the code isn’t doing anything truly wacky.
User facing apps are harder to validate, sure, but you can easily have an LLM looping looking for errors, log issues, etc.
For home apps, this is more validation than you would have personally done
Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.
We ended up building our own test platform to reduce the human bottleneck in the verification loop.
Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.
We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.
Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.
While doing local development where your environment is already setup and dependencies are installed doing something like this seems viable after the initial effort.
But how would this work in a software-factory setup, where coding agents run on fresh VMs or when using something like Claude Code on the web? How do you quickly recreate the required environment and dependencies there?
That’s essentially what we do. The agent writes the browser tests (and the other tests), we just review the replays of them afterwards from the test suite.
I'm building an orchestrator just for the "build" part of building.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
I think vision models will come more into play for validating things. It’s the most like consciousness, and less like - as you put it an LLM validating its own assumptions.
It’s at least an independent way of analyzing the work (as glyphs and images).
You can test for test vacuity with mutation testing, using tools like Stryker (for JS). In my experience this significantly helps models to write tests that actually test for correctness rather than aligned assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.