We ran 24 API calls through Claude Sonnet 4.6 - same questions about code changes, same model, same temperature. One gets git diff output. The other gets structured JSON from sem diff. Results: 95.9% accuracy vs 41.5%.
The failure modes are systematic, not random:
1. git diff has no concept of "entity." Ask an agent to count added functions and it counts + lines instead. On one commit it reported 238 adds - that was the number of + lines. Actual: 32 entities.
2. No way to distinguish add vs modify. A modified function shows +/- hunks, same as a new function in a changed file. The model listed 9 "added" functions - 4 were actually modified. Precision: 55.6%.
3. No entity type vocabulary. Asked for entity type counts, the model returned {"file": 11}. It counted files because line diffs have no AST. Truth: 15 functions, 12 interfaces, 3 variables, 1 class.
4. Config files are invisible. JSON/YAML changes are just +/- key-value lines. The model doesn't classify them as entities. Missed package.json entirely.
5. Large diffs get truncated. A 3,905-line diff hit the 100KB cap. The model saw partial context and found 25/67 functions (37% recall). Structured JSON is compact enough to fit - 64% recall on the same commit.
sem is a Rust CLI (30ms, single binary) that sits on top of git. It uses tree-sitter to parse your code into entities - functions, classes, properties - then does three-phase matching (exact ID, content hash, fuzzy similarity) to classify each change as added/modified/deleted/renamed.
Output is JSON. Pipe it into your agent, CI, or whatever. git stays your source of truth - sem just reads from it.
Hey i have been fed up with using coding agents on my local machine, but what I am looking for is an interface that can give the user an option to run on multiple sandboxing solutions. Do check it out.
Appreciate the honest feedback on the website - hearing this loud and clear from multiple people. The Product Hunt page definitely does a better job explaining it simply.
Re: AMD - I'd love to chat about this. We're running on Modal right now which uses NVIDIA, but supporting AMD would be valuable especially as we scale. What part of the stack do you have running? Would be interested in exploring a collaboration.
We've written a cuda to amdgpu compiler. It should be possible to persuade someone like Modal that this is wonderful. So far people largely decline to believe that it can be done at all :)
If you've written vaguely reasonable cuda, feed it to our nvcc binary and away you go. It's more annoying with dependencies, e.g. first one builds cutlass with our "nvcc" and so forth. Just occurred to me while writing this that we could start hosting builds of popular libraries ourselves.
Good question, it handles all the infrastructure. GitArsenal runs on Modal's serverless platform, and we automatically provision the right machines for each repo's requirements(working on this feature btw). You don't need to configure GCP/AWS or manage any cloud resources, just point us at a GitHub repo and we handle finding the appropriate compute, spinning up environments, and running the setup
The unsolved part: reliable, automated setup that works across the long tail of repos without human intervention. Docker helps but doesn't solve it (many repos don't have working Dockerfiles, and building those is itself a setup problem).
but dockerfile is not the complete setup you need to configure a lot of other things, and in a lot of cases dockerfile is not even getting generated correctly, try out gitpod they generate devcontainer.json which is a standard format across different platforms, but generating a devcontainer.json or Dockerfile has never been easy for agents specially when the repositories are complex.
The failure modes are systematic, not random:
1. git diff has no concept of "entity." Ask an agent to count added functions and it counts + lines instead. On one commit it reported 238 adds - that was the number of + lines. Actual: 32 entities.
2. No way to distinguish add vs modify. A modified function shows +/- hunks, same as a new function in a changed file. The model listed 9 "added" functions - 4 were actually modified. Precision: 55.6%.
3. No entity type vocabulary. Asked for entity type counts, the model returned {"file": 11}. It counted files because line diffs have no AST. Truth: 15 functions, 12 interfaces, 3 variables, 1 class.
4. Config files are invisible. JSON/YAML changes are just +/- key-value lines. The model doesn't classify them as entities. Missed package.json entirely.
5. Large diffs get truncated. A 3,905-line diff hit the 100KB cap. The model saw partial context and found 25/67 functions (37% recall). Structured JSON is compact enough to fit - 64% recall on the same commit.
sem is a Rust CLI (30ms, single binary) that sits on top of git. It uses tree-sitter to parse your code into entities - functions, classes, properties - then does three-phase matching (exact ID, content hash, fuzzy similarity) to classify each change as added/modified/deleted/renamed.
Output is JSON. Pipe it into your agent, CI, or whatever. git stays your source of truth - sem just reads from it.
brew install sem-diff
Benchmark script + full results: https://github.com/Ataraxy-Labs/sem/blob/main/bench/agent-ac...