The Migration Assistant was mostly a greenfield project and the roadmap was steered directly by feedback we continually received during development. It was built on top of our existing CLI-based tools for migrations, but the web UI and step-by-step wizard with AI assistance was all brand new.
What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.
We're discussing internally what can be released in the open, but nothing to share yet.
The closest thing we have is a fuzz testing framework that generates random schemas, drives the Migration Assistant end-to-end using Playwright, has an LLM judge the results, and files issues back into the hospital. It’s run manually, not after every merge.
Funny, right now we have a super bare bones of something like that. When the Chief (a human) makes a decision that generalizes (for example, a project-wide best practice), it goes into a precedent log. Before an agent escalates something to the Chief, it has to check the log, and if a precedent applies it cites it and follows it instead of escalating. It’s been one of the ways to cut down escalation noise but it definitely could be more mature.
Fair, the codebase might be more accurately described as the patient and the bug is the ailment it's being treated for.
The original idea of the metaphor is that a "github issue" is a patient that enters the hospital, gets "treated" (by being resolved), then leaves the hospital. The metaphor gets mixed up when we start using the word "bug" and "github issue" interchangeably.
Thanks. Yeah I agree that it’s not always a good fit. There are plenty of places where we bent the metaphor or just ignored it; like for example I also added a "test classifier" checklist that is really just about software engineering practices with no attempt at connecting it to doctors.
As we consider bringing this pipeline into larger repositories with more developers, we have discussed whether it's time to drop the metaphor. The last thing we want is for someone new to a project to have to learn the inner mechanics of the medical industry at the same as figuring what the hell is happening in a gigantic codebase.
Yes, the codebase still allows you to make a PR the normal way, and getting another engineer to review. That is, a human could make a PR totally by hand or by steering a Claude session on their machine.
I still find myself taking that approach fairly regularly. For most framework-level changes, or changes to core packages, I think the added scrutiny and deliberation is valuable. But for "leaf" components that don't have many dependents, using the hospital to move rapidly has been invaluable. I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
I like the control-group idea. It’s close to something we’re already building, where a human can label an issue to opt-in to write the plan or the code themselves. Routing a fixed % of issues that way would give us a baseline we don’t have right now.
To be clear, when I said "can you", I didn't mean "does the system allow this", more like "are you, the person I am speaking to who is the architect of the system, confident that you could make a fix to this system without the LLM's help?".
> I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
Yeah, this is kind of what I meant. I think in some future, these are going to be really valuable for maintainers.
Rather than per-day costs, I think it might be more meaningful to look at per-result costs. It took us about 9 months (and $160k using our napkin math) to add support for Oracle. Adding Db2 support using this agent pipeline took 2 days and $4,172 in tokens.
One caveat to the number for DB2 is that it doesn't consider review costs. We weren’t reviewing the hospital’s code during that sprint. Since we’re still confirming correctness for Db2, a more accurate comparison would add the human time to verify this.
I’m not going to claim this tripled our revenue. But revenue is also a function of time to deliver too. When a missing feature in the migration tool is what’s blocking a customer’s migration or a deal, shipping it in days instead of months can move revenue directly if it means the customer doesn't choose a different alternative.
Thank you for the reply. My question is - will you be able to charge the same amount for work done in days instead of months? Or even a proportional amount?
If its work that was blocking an existing deal, I can see this playing out in your favour. Whereas, if you were trying to negotiate a deal where you wanted the customer to pay x$ for a feature you have yet to develop, you might now have to charge lesser.
The closest thing we have is if a merged change has to be reverted, the Safety Department holds a Morbidity & Mortality conference and writes up what went wrong. A patient that needs three or more rounds of rework gets one too. There have been seven reverts so far, so not many funerals.
Sorry about that. The repo is internal to Cockroach Labs for now. It was an issue about adding a progress rollup comment to decomposed parent issues. We’ll get the post updated.
I like this. Today the parent issue closes automatically once every child is checked off, and nothing re-reads the parent’s original ask against what actually shipped. There is a deferred-scope auditor that catches the deferrals an agent admits to. Your "verify node" would catch the ones it doesn’t, which is the more dangerous kind. I think it would fit naturally as one final child per decomposed parent.
What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.
We're discussing internally what can be released in the open, but nothing to share yet.