Finding similar code is something different than deduplication, even when the final goal looks similar.
Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.
Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.
I don't understand what you mean with your first sentence. Both SemHash and Slopo are deduplication libraries right? Regardless, finding similar code is the core functionality that enables (semantic) deduplication.
I also don't think the backend is the "easiest part" here, a lot of the scalability lives there, which is important for monorepos, or cases where you want to deduplicate across projects. For example, from looking at the implementation, you use exact brute-force similarity search (comparing every item to every other item) which is an O(n^2) operation. It also allocates large dense similarity blocks in memory, so memory use won’t scale well either.
Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents.
Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.
"brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.
From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.
"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak.
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.
To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.
"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.
One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.
1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.
Fair, many tools trade off cost for performance/adherence. So far our experience is good with Semble (at least with Anthropic/OpenAI models), but if you have any feedback feel free to reach out. We're working on evaluations for this as well, though it will take some time as this is much harder than benchmarking retrieval quality.
There's a "semble savings" command you can run to see token savings. It varies heavily based on the repo: in general, the larger the project the larger the savings. Output code quality is something we're still trying to measure, but it's much harder than measuring retrieval quality.
Gotcha. Might be interesting to show some plots of average/median/max input token savings per query as a function of repo size if you're going to do some more comparison testing. The efficiency gains are compelling, but I'd want to see the magnitude of gains as well to get a full picture. Regardless, cool project and I'll check it out for myself
Hey, this skepticism is fair and we share it, which is why we don't claim end-to-end agent improvements since we haven't measured those (yet). The benchmark we published measures retrieval quality and token count during search, not overall agent performance. We are working on agent-level evals, but those are unfortunately much harder to get right. However, we do believe that Semble makes agents better based on our own experience of using it for the past months while in development (or at the very least, cheaper).
> We are working on agent-level evals, but those are unfortunately much harder to get right.
It's unfortunately a nearly impossible task, as the models change regularly (without letting you know), so you have a moving (invisible) target that's 1) hard to test exhaustively, and 2) very expensive to test with any low margin of error.
This is why no one does it and just makes broad sweeping unverified claims instead.
If you figure out how to do it... You should probably just get a job at Anthropic or OpenAI and make $2M+ per year...
Hey, what's the issue with a disk search-lib in python specifically? The library is extremely fast. Yes, we could probably squeeze some more performance in Rust, but that's not our native programming language, so we opted for doing it correctly rather than use a language that we don't understand well enough.
In theory maybe, but in practice it hurts more than it helps I think. Irrelevant context makes the model more likely to reason from the wrong code (and it's slower and more expensive).
Hey, thanks for the detailed feedback. For the bug, would you mind opening an issue with your setup details? This is definitely something we want to investigate and fix. The multiple queries thing is really good feedback, thanks for that, we'll update the prompt/instructions to prevent this from happening and we'll try to add some tests for this. The external connection errors during install are uv fetching deps from PyPI I think, those should not be the reason it's hanging.
The 98% is vs the grep+read loop, not grep output alone. When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience. If you're reliably getting agents to do "grep -C N" and stop there I'd genuinely be curious what your setup looks like, because I think the quality of the results is just too low to serve as useful context.
> When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience.
Depends on the size of the project and specific files. I have definitely seen agents make smart use of pi's "read" tool, which can take an offset and line limit (or defaults to a max 2000 lines/50KiB if the model doesn't specify). The bash tool also has the same max output, so if a model decides to cat instead of using the read tool it still wont blow out its context window with a single large file read.
But this sort of thing is going to vary with harness, model, project, and whatever the RNG delivers for the day.
