We assume LLMs can always be fooled, so the judge is just one step of the verification process. The deterministic layers, per-tool db roles, read-only-by-default integrations, query timeouts, are the ones an admin can configure. The judge's rules aren't exposed to admins today. I like your idea of letting admins add their own rules or knowledge to the judge, we should add that.
One of the core issue here is LLMs are probabilistic not deterministic, even with set of rules or context , llms are inherently built to make best guess, so my question is how you are ensuring that LLMs are following rules?
You're right, and we don't. The judge is best-effort screening not enforcement. Enforcement is deterministic with per-tool Postgres roles, proxy-injected creds, host allow-list, human approval on destructive writes. Those hold whether the judge is right or wrong.
Thanks! We believe the building aspect is already commoditized and we're experimenting with leaning into this with BYOA (bring-your-own-agent). Our bet is that the labs will stay focused on the model and won't do the more unglamorous governed deployment layer needs (credential scoping, per-tool db, audit, etc), but time will tell.
Don’t know if anyone else is feeling this, but I’m getting a sense they’re gona have to go in to this layer. Not enough money to warrant the investment in the model only approach…
Particularly if LLMs plateau and intelligence becomes essentially commoditized. If they can't compete on models alone they'll start moving more and more up the stack.
Thanks! We've actually have been experimenting with the config generation approach as well. One tradeoff we noticed was it turns the system into more of an app builder where the ceiling of what you can do is lower compared to a coding agent. But it's much faster than codegen. How dynamic have you found Lowdefy compared to codegen?
Thanks! Agree that for one-off questions, chat is the right interface. We’ve found apps win when the workflow is repeated across a team and they almost become “templates” that seed ideas for other coworkers.
Hey! I built this because everyone's writing CLAUDE.md files now but nobody knows if theirs actually works. The research is contradictory too, one paper says they hurt performance, another says they help. So I made a tool that just measures it on your own repo using your own PRs and your own test suite.
Turns out it's not often you can point to a single markdown file and say "this made the agent 27% better at resolving real tasks." That's what we saw on our production monorepo. I imagine this as a way for teams to actually make their agents write better code instead of guessing.
So by default it pulls out recent PRs, grabs only the tests that were commited and then checks to see how well an agent with different claude.mds can finish the test suite?
