No, they’re aware they don’t possess copyright on the source code because it has not one scintilla of human effort, so keeping it secret is the only legal way to protect it! :)
these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
Yeah, but none of those builds should have seen the light of day, given all the problems that were found.
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.
This is like paying for a pentest of a treehouse, and claiming that this implies the updated treehouse with a working padlock instead of a novelty lock is in the same league as a bank or a data center.
It makes mongodb from the report a decade ago look good.
There's no prize for participation, and they very clearly didn't do even the most rudimentary testing before paying for an in-depth third party analysis. I'd say that this reflects a profoundly childish approach to software, but that is unfair to children: I remember how much testing I did of my crappy game before I submitted it to STFormat when I was 12.
Most Jepsen tests leave me with the impression "databases are damn complicated and difficult to get right". My thoughts with this particular report are "I'll never ever touch this DB if I can help it".
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
I was referring to this claim from before the Jepsen report, which they have of course now quietly removed:
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.
Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?
I mean did you see the "primary key equality" thing? Holy shit.
Redesigned by whom? Given how bad your LLM was at bugfixing in the releases tested by Jepsen, I would expect the redesigned version to bring back all those bugs and then some.
I commend Orkes for funding a Jepsen report knowing that it could be published this harshly. I question the wisdom of doing so, but it at least speaks to the team's integrity. Building databases is very hard; I know this from personal experience.
Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.
I decided to dig into these claims a bit.
> Its also faster than Redis for KV
ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.
- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.
- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.
- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.
- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.
> also numbers show this should be top on clickbench
Would this claim survive after validating query results?
Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.
The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.
SDKs are MIT license CLI tools and Studio will be also be MIT License
Server is not open source for now but free for developers
reply