> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Fortunately the list of VCs funding this with millions and clearly zero due diligence is easy to find. I will add them to the address group for my next Nigerian Prince AI startup email distribution list.
They all got played by some ai dunning kreugers. The investors might not know it but anyone who knows anything about dbs knows that failing a jepsen test this badly is not the start of something. it's the end of something.
Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.
Go figure one of the founders is on openais board. Yikes...
I guarantee you that every investor knows they are not knowledgeable about technical details. So, they have a person on call that they’ve designated as their technical advisor (or even entrepreneur-in-residence, depending on how much the investor is paying this person).
The problem is there’s no incentive for this advisor to admit they are out of their depth because the money will stop flowing: some flavor of “if you can’t help me on this, then what am I paying you for?”.
Experienced investors, a.k.a. those who have previously been burned and learned from it will have multiple technical advisors categorized by their past career specializations.
The other investors will continue to rely on the advice of one jack-of-all-trades with questionable results.
I've been that advisor. You'd be surprised how often investors are straight-up conned by founders into becoming hype men for broken-by-design businesses as soon as the advisors are out of the room.
That's not because investors are idiots. A little bit of it is because a certain amount of credulity/faith is a necessary skill in that business. But most of it is just because humans are fallible and trickable in general. Advice from an expert can only help with that so much.
I'm part of the ParticleDB team. We took these findings seriously and have spent the past few months addressing them and strengthening our testing. We now run sustained Jepsen workloads daily alongside our own test suites. We've also narrowed the supported transaction modes to focus development and testing on core correctness.
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
> We now run sustained Jepsen workloads daily alongside our own test suites.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
What kind of introspection did you do on the root cause of those issues? "Data from deleted table reappears in new vaguely-similarly-shaped table" or "simple row write just doesn't happen" are bugs that are quite impressive to accidentally happen!
How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?
"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.
Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.
I would highly recommend leaving the industry. You are not at the caliber required for this kind of work, and you are not aware that you are not at the caliber required for this kind of work. You bring shame to our profession.
Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.
I mean I assume that that's where, say, 3.3.2 came from; the database grows _new_ bizarre brokenness as stuff is fixed.
I'm genuinely kind of fascinated whether there was a _design_ here, or if someone just said "claude, postgres but faster pls". Some of the issues are truly incredible.
If you took the findings seriously why didn’t you take the outrageous claims you were making in your own copy seriously? Why couldn’t you provide cogent analysis of these issues during the process? How did you hack up your db so bad in the first place?! This thing looks so bad bro I can’t help but think this kind of assertion only adds fuel to the fire.
> we're working on benchmarking now as well.
But you already said you were 1000x faster than Postgres??
Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.
No one in their right mind will use this product ever.
I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.
Its not happening, at least not with LLMs, maybe some future technology will be better with that, but not language models. Writing such a complex software as database engine requires strategy, awareness and rigor. It is somewhat similar to winning a sweaty game of chess with an experienced opponent. LLMs are great with broadness of their knowledge, matching patterns and synthesising that knowledge, but they fail with strategy and awareness part. And this is why they are bad in chess.
It is combinatorial explosion of possible states (not as in memory state of program, but the differing visions of the next state of architecture). Plus high impact of mistakes, suboptimal solutions and randomness when switching between states.
How does this make the case for humans forever exceeding LLMs in planning and architecture? It doesn’t explain why a neural net couldn’t learn to do better than humans on this task at all. It certainly doesn’t point out any deficiency of attention, transformers, layers, encoders, decoders, or how the layers or their inputs/outputs are set up.
Yeah, good question, thanks. Llm reasoning is done exclusively in the way of symbolic manipulation. Human reasoning is done not in symbols but in mental models that are hard to explain for today science but we have evidence they are somewhat efficient for producing correct programs, at least after sufficient training. Ron's post makes the case to explain that symbolic reasoning is not sufficient for producing correct programs. Hope this helps.
> Llm reasoning is done exclusively in the way of symbolic manipulation.
No, LLMs are not a classical symbolic system applying explicit rules to symbols.
The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too. While that suggests neither humans nor LLMs can reliably solve every planning or verification problem, it says nothing about which will do better on the software problems people actually encounter.
It certainly does not present a fundamental problem with LLMs as they exist today.
> No, LLMs are not a classical symbolic system applying explicit rules to symbols.
I do not mean that LLM is a classic symbolic system that generates symbols according to rules,
but instead it's a system that generates symbols according to recorded probabilities. The focus here is on symbols: the LLM reasoning starts with <think> token, ends with </think> token, and everything in between is symbols (tokens). Quite different from how human brain works, where symbols are not that much involved in reasoning.
> The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too.
Yep, however I would like to assert that today we have evidence of systems architected by humans and behaving correctly (like Linux kernel), while there are no evidence of systems architected by LLMs that behave correctly. Even the systems which are architected by LLMs with human assistance (such as Claude Code or Codex) routinely have horrendous bugs (you're welcome to search "Claude Code Bug" and "Codex Bug" on this very website).
> It certainly does not present a fundamental problem with LLMs as they exist today.
The article was created 10 years ago that is why it does not address LLMs. However it asserts that mathematics modeling is not that helpful for writing correct programs. As long as LLM reasoning is closer to mathematics modeling than human mental modeling, LLMs will be worse at writing correct programs than humans. Some future technology less focused on symbolic reasoning could change that, but today it's not on sufficient level.
Nobody cares about conviction. What can be demonstrated is what matters.
darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.
What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.
Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
Or try any novel architectural discussion and see for yourselves.
Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).
Talking about current LLMs while trying to assert future impossibility, and not mentioning an actual architectural problem, is giving a hint of not seeing past your own nose tbh
Your reading skills are commendable, especially since you "found" an assertion of future impossibility. Me having explicitly mentioned the future as "up for debate" makes your reading comprehension even more impressive.
Unable to see the architectural problem mentioned is just icing on the cake.
Wrong. The entire arc was on database and novel architectures. Let me point you to the things you missed both on the example and the context.
>> Writing such a complex software as database engine requires strategy, awareness and rigor.
> Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.
> Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
I find it unlikely, especially due to the numerous pathetic suggestions LLMs keep offering me on the architectural discussions I engage in.
Verbal abuse is what makes it useful to me. A punching bag where it infuriates me making me lay down the how and why its suggestions are stupid. This results in progress.
I guess we would have to wait and see. Maybe LLMs do get better at architecture in the future. But seems unlikely to me without expert supervision.
I was thinking that this is exactly what ParticleDB is: a random amateur vibe coding a DB. I wouldn't count on things changing much in a couple of years, people will still be people.
Amateur vibe coders who raised a $60M Series B a few months ago.
I'm genuinely not sure who comes out looking worse - the coders for their reckless amateurishness, or the VCs for providing a misleading veneer of professionalism.
It's the old infinite monkeys typing on type writers except now it displaces an entire industry and the only people profiting are the ones holding the tech that powers it.
these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
Yeah, but none of those builds should have seen the light of day, given all the problems that were found.
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.
This is like paying for a pentest of a treehouse, and claiming that this implies the updated treehouse with a working padlock instead of a novelty lock is in the same league as a bank or a data center.
It makes mongodb from the report a decade ago look good.
There's no prize for participation, and they very clearly didn't do even the most rudimentary testing before paying for an in-depth third party analysis. I'd say that this reflects a profoundly childish approach to software, but that is unfair to children: I remember how much testing I did of my crappy game before I submitted it to STFormat when I was 12.
Most Jepsen tests leave me with the impression "databases are damn complicated and difficult to get right". My thoughts with this particular report are "I'll never ever touch this DB if I can help it".
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
I was referring to this claim from before the Jepsen report, which they have of course now quietly removed:
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.
Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?
I mean did you see the "primary key equality" thing? Holy shit.
Redesigned by whom? Given how bad your LLM was at bugfixing in the releases tested by Jepsen, I would expect the redesigned version to bring back all those bugs and then some.
I commend Orkes for funding a Jepsen report knowing that it could be published this harshly. I question the wisdom of doing so, but it at least speaks to the team's integrity. Building databases is very hard; I know this from personal experience.
Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.
I decided to dig into these claims a bit.
> Its also faster than Redis for KV
ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.
- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.
- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.
- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.
- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.
> also numbers show this should be top on clickbench
Would this claim survive after validating query results?
Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.
The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.
I’ve come to learn an interesting dichotomy in software:
Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.
Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)
ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.
As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.
[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.
> Given the severity and extent of the safety errors we found, and the process which produced them, Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Consider checking and/or rebuilding database state from more reliable sources, if available.
I guess that's what happens when you vibecode a database...
MongoDB had consistency issues when running as a distributed system, that's expected if you read Jepsen's reviews. Meanwhile, ParticleDB had bugs like this when running as a single node:
root=> CREATE TABLE t
(a int primary key, b int);
root=> INSERT INTO t VALUES (4, 5), (5, 6);
root=> SELECT * FROM t WHERE b = 4;
a | b
---+---
4 | 5
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.
It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?
My best guess is their board/investors forced them to.
In this theory, without evidence just like their db claims, they were looking for external evidence to shut it all down or force replacement of key personnel.. And Aphyr understood the assignment.
I'd assume that no-one who works there really knows anything about databases; they may genuinely have thought that it worked. A lot of super-AI-pilled people seem to be deep into this "expertise doesn't matter any more" belief system.
And no doubt they wanted a headline to help boost the Anthropic IPO (I'm kidding but knowing the crypto crowd and the overlap with the AI crowd, who knows)
> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
The AI boosters keep complaining that their work isn't evaluated on technical merit. Thank you for providing strong evidence that "the abstraction layer just moved up" mindset and vibing all your code, infra and culture does not lead to high quality software.
Reality is probably closer to “moving the abstraction layer up is a nontrivial process that people are still figuring out and it’s easy for LLMs to produce garbage code that they tell you is good”
(Edit for clarity: this is quoted from the linked article)
You could write predicates over non-existent columns:
root=> SELECT * FROM t
WHERE definitely_not_a_column = 4;
a | b
---+----
4 | 5
(1 row)
Or ask if 4 = 5
root=> SELECT * FROM t WHERE 4 = 5;
a | b
---+----
5 | 6
(1 row)
Good on them for being good sports I guess, but after reading this I would never touch a piece of software built by anyone involved with this company. The sheer hubris of their claims combined with the specific reality of their broken garbage software (with the contradicted marketing claims and hallucinated documentation) is literally breathtaking.
It must have been darkly amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so-slowly start to realize their software quality is not quite what their glazing agents had been telling them and that what they have on their hands is basically sub-mongo-quality slop. What a world we live in, huh?
I won't push back on it being garbage, but I would suggest that if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork, e.g. [1]
I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL. Your example queries remind me of that.
> WHERE column = NULL and then query WHERE column <> NULL
With respect, your statement seems to indicate that you simply haven’t heard of or don’t understand the three-valued-logic semantics behind SQL.
NULL means what it means and is conceptually foundational to SQL. Comparing it to anything returns UNKNOWN, the “third value” besides true and false. And WHERE needs a TRUE predicate. So your implied fault that they “have no problem“ really is just a matter of you not understanding the tool you are complaining about.
To match against the NULLness of a value you must use special purpose operators. In this case
> therefore that whole WHERE clause doesn't exist either
No, therefore the condition is unsatisfiable and equivalent to WHERE FALSE. And that's letting you ignore the SQL spec, which says the query is invalid and should return "Column not found" because the column name cannot be resolved.
1) It would be no more fucked up than anything else that has been grandfathered into our tools.
2) If it already had mass adoption in the market, people would jump onto this thread to explain to me that I just don't understand the brilliant semantics behind language of why 4 = 5.
I spent more time than I’d like to admit trying to understand your comments here. I’m convinced you are a very odd person with a confused mind.
Brilliant semantics? No one said anything normative. You’re arguing with yourself on your own non-sequitur. SQL is SQL. It’s extremely flawed but who freaking cares, that’s why ParticleDB shouldn’t have tried to implement SQL let alone full postgres compatibility when they clearly are not equipped to do so! Your comments about how strange and non-intuitive SQL are true but irrelevant.
> I won't push back on it being garbage, but I would suggest that if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork
No. No-one would ever accept a database with _this_ level of brokenness. Like, this makes early Mongo and noughties-era mysql look _amazing_.
> I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL.
You may not like it, but that is entirely deliberate and internally consistent. Blame the people who came up with SQL in the 70s; far too late to change it now.
I am rather receptive to the argument that SQL is a fractal of bad design. But complaining that SQL engines implement well known SQL semantics according to spec is silly.
it's unfortunate that you are getting downvoted for having an actual conversation rather than just piling on these guys. but there are also a lot of places where people have said "everyone is using LLMs" and also "this thing about the job being requirements is a myth, because lots of requirements are ridiculous" has also been said, so i don't know. what do people want? a lot of these bullshit startups to fail, which is funny, but it's not good.
And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.
You know how some people think that LLMs are magic? I assume that's more or less what happened here; the database was a result of prompting the finest LLMs, so how could it _not_ be perfect?
Yeah, wouldn't touch anything this company makes with a ten foot pole.
Maybe you don't know too much about engineering, but things happen, bugs are possible. Rome was not built in a day. All systems have bugs. The source being closed, with no QA, what do you expect. Maybe you don't remember Postgres in the early 2000s.
I remember Postgres in the early 2000s, and it didn't have anywhere near this level of problems. It had its issues, especially in terms of performance, and there were of course occasional correctness bugs that tended to turn up when you were doing something unusual. But you could certainly rely on basic SQL operations to work correctly.
> Maybe you don't remember Postgres in the early 2000s
Postgres was already mature by the early 2000s - it was started in 1986 by a veteran who had written Postgres' spiritual predecessor in the 70's (Ingress). So no, Postgres didn't have amateur-hour data loss bugs in the early 2000s. What Postgres was missing out of the box were some features, like replication.
> Maybe you don't remember Postgres in the early 2000s.
Not even the grungiest version of MySQL 3.23 running on some shared hosting provider in the early 2000s would behave as immediately illogically as the database under test here.
I remember Postgres in the early 2000s. And early Mongo. And many others.
And bugs like “WHERE 4 = 5”, or selecting from nonexistent columns, or selects permanently modifying table behavior, or any of a dozen others that Jepsen found … didn’t happen in released databases. Even young ones.
Prior to the age of LLMs, that level of bugginess wasn’t “everything has growing pains”. That’s the kind of thing previously associated with poor undergraduate CS homework or “the CEO’s kid wrote our database” corporate horrorshows.
> with no QA
That’s not immaturity. That is a choice. A “the car we are selling does not have airbags, seat belts, or brakes” kind of choice.
> Maybe you don't know too much about engineering
Save it. You have no idea what you are talking about.
Imagine asking for a review before checking with the brutally powerful agents we have today all the low hanging fruits. And, at the point we are, you can likely go at incredible levels of automated bugs seeking and do the whole test in house.
It is months that I find subtle bugs far more complicated than the most immediately findable issues in the report. Together with the rest of the world. So there must be something wrong with their AI usage.
I do not know in what order Kyle et al. wrote the report, but as I progressed through different parts I think I could detect instances of an increasingly fatigued/disheartened/unhinged tone--without compromising the professionalism we expect from Jepsen analyses, of course. These moments of humanity brought me joy.
More cynically, I suspect that many low-quality database projects succeed or fail based more on marketing and hype than robustness. A dishonest product could simply take the line "engineers at Orkes were able to produce a highly sophisticated database system" from the report out of context, quote it on their homepage attributed to aphyr/Kyle/Jepsen, and reap short-term rewards and popularity. I hope I'm wrong.
I think linking to TikTok videos saying “vibes? vibes!” in some of the links was a bit below the belt, but understand it takes restraint to maintain NPOV
I'd guess that all the restraint in this case has been spent on staying professional throughout writing all the individual bug descriptions. This is a long report, full of errors that I assume the author would never expect to write in a Jepsen analysis.
Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
They probably asked an AI to run the tests against their DB. And the AI said "yeah I ran it, all green!" (except with a lot more words and perfect grammar, and maybe a few section headings and emoji.)
Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.
Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.
The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.
A long time ago I worked with Aphyr to evaluate a database with Jepsen. He was extremely professional, and he found a few real issues which we resolved extremely urgently. We took things seriously and wrote a significant amount of testing ourselves, and by the time our engagement began, we were pretty confident. Overall it was a great experience and I highly recommend his services.
By comparison, this report paints an absolute clown show. The amount of anomalies is perhaps understandable (a single architectural flaw can have a lot of impact) but the fact that they found so many disparate issues is telling.
The fact that they then said they addressed the issues, but didn't, is about the biggest red flag imaginable. I would never let any client use this software under any circumstances.
To everyone pointing out that artificial intelligence is to blame here, you may want to read his takedown of MongoDB back in 2013: https://aphyr.com/posts/284-jepsen-mongodb
> have we actually ruled out human hubris and gross incompetence
it takes a lot of of human hubris and incompetence to think you can herd AI agents to squirt out a better version of a multi-decade project in weeks, without a single human in the loop. I can't imagine the levels of hubris in prompting "Make a faster Postgres in rust. Make no mistakes" and immediately uploading the result for Jepsen testing without bothering to manually kick the tires a bit to see if your DB is fit for purpose.
I see a direct line between this attitude and the Proton CEO deriding "loser thinking" in a subordinate that dares suggest the AI output have some oversight before going to prod; there exist people who are so AI-pilled they get lost in the sauce.
I am not sure I would want my database tested by a vendor who feels it their duty to provide color commentary on my software development methodology in a public report, no matter how rigorous their evaluation.
> vendor who feels it their duty to provide color commentary
Honestly? At some point there's a fiduciary-like obligation for it. What ParticleDB delivered is so far past good or bad, it's just straight ugly and then some. It's so poor that, if Jepsen couched it in any way, they'd be implicitly misleading their readers.
The principal values of things like the Jepsen reports is to the community of potential users/purchasers.
I agree with GP, that vendors should be somehow required or very strongly encouraged to participate in such audits. Not because they want to (as you say, most wouldn't like their dirty laundry aired thus), but because consumers deserve to know. U.S. government software groups don't like FISMA/RMF audits either, but they have to do them anyway.
I am of a contrary opinion. At some level of developer insolence, such commentary is valuable - especially if it needs to mention the core basics and foundations that keep the software world running, such as running existing Postgres test suites for a supposedly Postgres-compatible database. In the increasingly sloppified world, even such basics are being eroded - this being a proof.
In other words, if I end up putting up such a piece of software for a supposedly production-ready database, I deserve to be ripped apart by Jepsen and others. Somebody has to assert software quality; if it's not asserted internally, it must be asserted externally.
In yet another words, software development is a trust-based business. If you break it, such as by outright lying to people, then you deal with the consequences.
Aphyr (and/or his team) is the best technical writer in the biz. These Jepsen reports are always so clearly explained. This one is fun because the tone occasionally verges on disgust.
Is/was ParticleDB open source? There is a Github link on their website, however the only public repo in their org seems to be the Golang client library.
No, they’re aware they don’t possess copyright on the source code because it has not one scintilla of human effort, so keeping it secret is the only legal way to protect it! :)
As any other project out there, we have bugs, but we solve them in public.
We have more that 30k unit tests in place, ha-integration tests, ha-resilience tests, chaos tests and more.
Coverage is almost at 80% (https://app.codecov.io/github/ArcadeData/arcadedb) Are all the tests green? No, we have some flaky tests and we do our best to fix them form time to time.
If something isn't working on your side, please file an issue
I wonder what the internal testing pipeline looked like before this. A codebase with that many fundamental issues suggests either no integration tests were written or they were all mocked out to pass green. It seems strange that a team would market a system as formally proven without actually running the workload through even basic consistency checks themselves first.
That part is the weird bit to me. If the existing harness breaks right away, that's the signal to stop and fix things, not to book an outside audit. Running the suite in CI on every build would have caught a lot of this.
> Running the suite in CI on every build would have caught a lot of this
yeah, provided:
1. test cases are being designed to cover functionality
2. test cases are implemented correctly - i.e. they actually fail if the implementation is defective
3. the tests aren't all commented out
4. the test suite fails if a test fails
5. CI actually detects test suite failure
6. failing tests block changes from being merged (apart from initial triage to stabilise the suite)
Presumably a few of these things weren't being done -- I'd bet 1 & 2 were skipped or implemented so poorly that they were worse than having no tests at all -- checking nothing but giving a false illusion that some kind of testing had happened.
Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.
In that case Ant/OAI should give Aphyr a blank check and infinite access to tokens to build the Mother-of-all-Databases. Anthropic has a few days before IPO day, they better get to it!
Congrats to the particledb team for getting this far and thanks to @aphyr for the analysis.
It takes gumption to build a new storage engine. I remember when etcd[0] first made it's way into @aphyr's cross-hairs. We took all of the input very seriously because we knew that a data store which is untrustworthy is useless.
Resolving these issues likely won't be easy, but if you can get to the point where you walk away from Jepsen unscathed the trust in your software will be greatly amplified.
Right, when Rachael Gunn (Raygun) embarrassed Australia and the whole of breakdancing on the olympic stage, the nice thing to say would have been "congrats on getting this far, fixing the issues won't be easy, etc..." but what really should be said something like "I'm not sure how it got to this point, but everything about this is wrong, and you should probably focus on something else".
Reads like investor cope. And the OG etcd was a disaster that needed better engineers to completely rewrite. The surviving project is an entirely different lineage.
Brutal!
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Fortunately the list of VCs funding this with millions and clearly zero due diligence is easy to find. I will add them to the address group for my next Nigerian Prince AI startup email distribution list.
They all got played by some ai dunning kreugers. The investors might not know it but anyone who knows anything about dbs knows that failing a jepsen test this badly is not the start of something. it's the end of something.
Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.
Go figure one of the founders is on openais board. Yikes...
I guarantee you that every investor knows they are not knowledgeable about technical details. So, they have a person on call that they’ve designated as their technical advisor (or even entrepreneur-in-residence, depending on how much the investor is paying this person).
The problem is there’s no incentive for this advisor to admit they are out of their depth because the money will stop flowing: some flavor of “if you can’t help me on this, then what am I paying you for?”.
Experienced investors, a.k.a. those who have previously been burned and learned from it will have multiple technical advisors categorized by their past career specializations.
The other investors will continue to rely on the advice of one jack-of-all-trades with questionable results.
I've been that advisor. You'd be surprised how often investors are straight-up conned by founders into becoming hype men for broken-by-design businesses as soon as the advisors are out of the room.
That's not because investors are idiots. A little bit of it is because a certain amount of credulity/faith is a necessary skill in that business. But most of it is just because humans are fallible and trickable in general. Advice from an expert can only help with that so much.
I'm part of the ParticleDB team. We took these findings seriously and have spent the past few months addressing them and strengthening our testing. We now run sustained Jepsen workloads daily alongside our own test suites. We've also narrowed the supported transaction modes to focus development and testing on core correctness.
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
> We now run sustained Jepsen workloads daily alongside our own test suites.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
> I'm part of the ParticleDB team.
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
We have extensive unit tests and integration tests that we run in our build process as we test the db.
What you're missing is credibility.
Show us.
both "extensive" and "test" are the load-bearing terms there
at this point even “run” is in question
What kind of introspection did you do on the root cause of those issues? "Data from deleted table reappears in new vaguely-similarly-shaped table" or "simple row write just doesn't happen" are bugs that are quite impressive to accidentally happen!
How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?
"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.
Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.
I would highly recommend leaving the industry. You are not at the caliber required for this kind of work, and you are not aware that you are not at the caliber required for this kind of work. You bring shame to our profession.
I wish more of us had the courage to speak up and say things like that. thank you
I second this. I wish more people spoke up against gross negligence abd incompetence like this.
Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.
Psychosis?
I struggle to imagine how an organization which is capable of taking things seriously would have produced software with these issues to begin with.
Hi Claude, can you fix all the Jepsen findings and push to prod. Thanks.
There is no doubt in my mind this is exactly what they did after this was published.
I mean I assume that that's where, say, 3.3.2 came from; the database grows _new_ bizarre brokenness as stuff is fixed.
I'm genuinely kind of fascinated whether there was a _design_ here, or if someone just said "claude, postgres but faster pls". Some of the issues are truly incredible.
you joke but that sounds like how many of the fully AI-psychotic shops operate
Where is the TLA+ proof?
TLA+ does not do what you think it does.
Go on, you're not saying anything yet.
I dont do consulting for free. I set you up already on the path of enlightenment :-)
If you took the findings seriously why didn’t you take the outrageous claims you were making in your own copy seriously? Why couldn’t you provide cogent analysis of these issues during the process? How did you hack up your db so bad in the first place?! This thing looks so bad bro I can’t help but think this kind of assertion only adds fuel to the fire.
> we're working on benchmarking now as well.
But you already said you were 1000x faster than Postgres??
Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.
No one in their right mind will use this product ever.
should you have done that in the first place? its a database engine
"a library of confusion" is exactly what you should expect from a 100% vibecoded product as complex as a database.
Writing databases is an EXTREMELY fun learning endeavour but it is truly difficult to do correctly.
Once I started writing one (I could say I still am) because I was unhappy with most of them and also, to see how hard it could be
Alas! It turns out it's really hard to do it right.
"But AI can oneshot Redis, so we can just vibecode it."
I wonder if it’s vibe coded by amateurs
I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.
Its not happening, at least not with LLMs, maybe some future technology will be better with that, but not language models. Writing such a complex software as database engine requires strategy, awareness and rigor. It is somewhat similar to winning a sweaty game of chess with an experienced opponent. LLMs are great with broadness of their knowledge, matching patterns and synthesising that knowledge, but they fail with strategy and awareness part. And this is why they are bad in chess.
I think it is probably going to happen with LLMs, and I'd bet on it.
As said: 1 - 2 years, not current ones.
Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.
What exactly is the architectural problem that has you convinced LLMs cannot learn how to strategize and architect well after enough RL?
It is combinatorial explosion of possible states (not as in memory state of program, but the differing visions of the next state of architecture). Plus high impact of mistakes, suboptimal solutions and randomness when switching between states.
Edit: I like this 10 years old blog post in regards to explaining the problem much better than I could do: https://pron.github.io/posts/correctness-and-complexity
How does this make the case for humans forever exceeding LLMs in planning and architecture? It doesn’t explain why a neural net couldn’t learn to do better than humans on this task at all. It certainly doesn’t point out any deficiency of attention, transformers, layers, encoders, decoders, or how the layers or their inputs/outputs are set up.
Yeah, good question, thanks. Llm reasoning is done exclusively in the way of symbolic manipulation. Human reasoning is done not in symbols but in mental models that are hard to explain for today science but we have evidence they are somewhat efficient for producing correct programs, at least after sufficient training. Ron's post makes the case to explain that symbolic reasoning is not sufficient for producing correct programs. Hope this helps.
> Llm reasoning is done exclusively in the way of symbolic manipulation. No, LLMs are not a classical symbolic system applying explicit rules to symbols.
The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too. While that suggests neither humans nor LLMs can reliably solve every planning or verification problem, it says nothing about which will do better on the software problems people actually encounter.
It certainly does not present a fundamental problem with LLMs as they exist today.
> No, LLMs are not a classical symbolic system applying explicit rules to symbols.
I do not mean that LLM is a classic symbolic system that generates symbols according to rules, but instead it's a system that generates symbols according to recorded probabilities. The focus here is on symbols: the LLM reasoning starts with <think> token, ends with </think> token, and everything in between is symbols (tokens). Quite different from how human brain works, where symbols are not that much involved in reasoning.
> The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too.
Yep, however I would like to assert that today we have evidence of systems architected by humans and behaving correctly (like Linux kernel), while there are no evidence of systems architected by LLMs that behave correctly. Even the systems which are architected by LLMs with human assistance (such as Claude Code or Codex) routinely have horrendous bugs (you're welcome to search "Claude Code Bug" and "Codex Bug" on this very website).
> It certainly does not present a fundamental problem with LLMs as they exist today.
The article was created 10 years ago that is why it does not address LLMs. However it asserts that mathematics modeling is not that helpful for writing correct programs. As long as LLM reasoning is closer to mathematics modeling than human mental modeling, LLMs will be worse at writing correct programs than humans. Some future technology less focused on symbolic reasoning could change that, but today it's not on sufficient level.
Thank you for the assessment and the link. I only read half of it as I was busy. Bookmarking it for later reading. Great read.
Yes. The combinatorial explosion is a great assessment of the "why".
But the problem is more nuanced than that because one has to assume that LLMs can reason accurately all the time which is demonstrably false.
The current LLMs cannot even state true sentences as true if they sound confident, without hedging.
Future LLMs may mitigate this, though it looks unlikely.
Nobody cares about conviction. What can be demonstrated is what matters.
darkwi11ow, in the sibling comment, gave a substantive response on the underlying mechanics with a link. Godel's incompleteness theorems also yield similar insight. But I want to shine some light on a different phenomenon.
What happens in the future is up for debate, but the current architectural reasoning capabilities of LLMs are pathetic.
Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
Or try any novel architectural discussion and see for yourselves.
Modern LLMs cannot even confidently state many true sentences as "maximally" true as possible without hedging and tone policing as if we were doing politics ;).
Hence an expert is required at every turn.
Talking about current LLMs while trying to assert future impossibility, and not mentioning an actual architectural problem, is giving a hint of not seeing past your own nose tbh
Your reading skills are commendable, especially since you "found" an assertion of future impossibility. Me having explicitly mentioned the future as "up for debate" makes your reading comprehension even more impressive.
Unable to see the architectural problem mentioned is just icing on the cake.
How do you do this?
Here let me point you to the assertion you made:
>> Its not happening, at least not with LLMs
> Correct.
Wrong. The entire arc was on database and novel architectures. Let me point you to the things you missed both on the example and the context.
>> Writing such a complex software as database engine requires strategy, awareness and rigor.
> Correct. Current LLMs are pathetic at architectural correctness without expert supervision at every turn.
> Try having a discussion on designing an architecture for a new kv engine with cache-oblivious algorithms that beats lmdb on generic random point-key lookups on a disk-backed mmap file.
> What happens in the future is up for debate,
Yes, but I said 1 - 2 years, not current LLMs.
My comment was to your parent comment :).
I find it unlikely, especially due to the numerous pathetic suggestions LLMs keep offering me on the architectural discussions I engage in.
Verbal abuse is what makes it useful to me. A punching bag where it infuriates me making me lay down the how and why its suggestions are stupid. This results in progress.
I guess we would have to wait and see. Maybe LLMs do get better at architecture in the future. But seems unlikely to me without expert supervision.
I was thinking that this is exactly what ParticleDB is: a random amateur vibe coding a DB. I wouldn't count on things changing much in a couple of years, people will still be people.
Amateur vibe coders who raised a $60M Series B a few months ago.
I'm genuinely not sure who comes out looking worse - the coders for their reckless amateurishness, or the VCs for providing a misleading veneer of professionalism.
I don't get how you can name your DB after particleboard, not the strongest material, and achieve hype ;-)
It's the old infinite monkeys typing on type writers except now it displaces an entire industry and the only people profiting are the ones holding the tech that powers it.
these were tests done on pre release builds and some experimental concurrency modes as well.... The engine has since been redesigned and all of these issues have been fixed..
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
> these were tests done on pre release builds
That's a convenient excuse, because now the website calls them "release candidate" builds, but as far as I can tell there was no such disclaimer back in May. As mentioned in the report, the Docker images were specifically advertised as being production-ready.
https://web.archive.org/web/20260518201315/https://particled...
https://web.archive.org/web/20260518200305/https://particled...
Yeah, but none of those builds should have seen the light of day, given all the problems that were found.
And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).
What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.
Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.
Because this is why companies pay for jepsen tests. Not because they enjoy to be destroyed in jepsen report, but to fix the issues.
This is like paying for a pentest of a treehouse, and claiming that this implies the updated treehouse with a working padlock instead of a novelty lock is in the same league as a bank or a data center.
It makes mongodb from the report a decade ago look good.
I'm not saying it is proof it is done. But most likely they are working on it
There's no prize for participation, and they very clearly didn't do even the most rudimentary testing before paying for an in-depth third party analysis. I'd say that this reflects a profoundly childish approach to software, but that is unfair to children: I remember how much testing I did of my crappy game before I submitted it to STFormat when I was 12.
They clearly aren’t the right team to be working on it. It’s best if someone else gives it a go.
Most Jepsen tests leave me with the impression "databases are damn complicated and difficult to get right". My thoughts with this particular report are "I'll never ever touch this DB if I can help it".
Yes they are complicated. But with enough rigor, structure and discipline, it is doable without breaking correctness guarantees.
Making it distributed opens a whole can of worms, which is where Jepsen shines.
[dead]
Ok sure, so an LLM told you all the bugs are fixed and it’s faster than Redis, just like how it told you before that it was formally verified?
Please try to resist the urge of being a meat proxy.
> before that it was formally verified
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
I was referring to this claim from before the Jepsen report, which they have of course now quietly removed:
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
Makes you wonder if they asked their agent to formally prove it with TLA+ and it just responded that it had performed the proofing.
The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.
Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?
I mean did you see the "primary key equality" thing? Holy shit.
/dev/null is also an extremely fast KV store.
Sure, it still has some bugs resulting in occasional data loss, but it is pre-release so that's okay. Look at the speed, people!
That reminds me of this classic https://www.youtube.com/watch?v=b2F-DItXtZs at 1m42s.
You just need to raise 50m to crack that pesky "consistent read after write" nut!
> and all of these issues have been fixed..
You can't be sure until they test it again
I look forward to the second report from Jepsen for the new implementation then.
I just wonder why, from all the things you can vibecode, you would pick a database if you don't know the first thing about database design?
> Its also faster than Redis for KV
No. Wrong. False. All performance numbers are plainly invalid. As my coworker says, performance numbers don't count if the code is broken.
Redesigned by whom? Given how bad your LLM was at bugfixing in the releases tested by Jepsen, I would expect the redesigned version to bring back all those bugs and then some.
I commend Orkes for funding a Jepsen report knowing that it could be published this harshly. I question the wisdom of doing so, but it at least speaks to the team's integrity. Building databases is very hard; I know this from personal experience.
Disclosure: I'm a software engineer on the core team at a different database company, but we focus on time-series use cases and do not compete in the ACID/OLTP space ParticleDB is targeting.
I decided to dig into these claims a bit.
> Its also faster than Redis for KV
ParticleDB's own benchmarks[0] come with several caveats. Much of the win over Redis seems to come from comparing ParticleDB on 48 cores to Redis's single execution thread, from what I can tell.
- On a single client in async mode, ParticleDB loses on writes at 0.86-0.90x and ties on GET.
- SET at 512 clients reaches 4.81x Redis, but that's about 18.7k ops/s per allocated core vs Redis's 186k on one thread. For comparison, GET at 512 clients works out to about 33.8k ops/s per allocated core vs Redis's 228k.
- Tail latency on SET is much worse, e.g. p99 of 49ms vs 5.3ms for async SET at 512 clients and 306ms vs 41ms for durable mode with pipelining.
- Each test was 8 seconds long on 100k keys, but the Jepsen report saw throughput decline linearly over an hour, even on small datasets.
> also numbers show this should be top on clickbench
Would this claim survive after validating query results?
Regardless, it's moot, because it doesn't survive on its face either. On hot runs, by my calculations from the numbers in the Clickbench PR[1], ParticleDB scores about 1.5x better than ClickHouse, however Umbra still scores about 1.5x better than ParticleDB.
The fact that `GROUP BY URL` (Q33/Q34) beats `GROUP BY UserID` (Q15) by 4x on a `c6a` is also puzzling to me. Only a handful of 100+ ClickBench results that I checked do this, and ParticleDB's ratio is the most extreme. It also speeds up only 2.2x when going from 16 to 192 vCPUs, while Umbra, ClickHouse, and ElosDB manage 7-10x. This makes me wonder whether the ParticleDB submission may inadvertently be using some precomputation/intermediate caching, despite final result caching seemingly having been disabled.
[0]: https://particledb.ai/benchmarks/key-value/
[1]: https://github.com/ClickHouse/ClickBench/pull/2952
https://github.com/ClickHouse/ClickBench/pull/2952
Previous attempts were strange, to say the least: https://github.com/ClickHouse/ClickBench/pulls?q=is%3Apr+par...
Even the PR description reads like it has been generated by an LLM.
What a waste of CO₂.
> Hey Claude, could you find some releases we can send to Aphyr for testing?
... executing SELECT * FROM releases WHERE 4 = 5 -- it's not just a slopdb, but TLA+ load bearing and 100% Jepsen ready
ParticleDB 1.1.4 ParticleDB 1.1.18.11.rc-22
2 rows returned
I’ve come to learn an interesting dichotomy in software:
Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.
Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)
ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.
As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.
[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.
> Given the severity and extent of the safety errors we found, and the process which produced them, Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Consider checking and/or rebuilding database state from more reliable sources, if available.
I guess that's what happens when you vibecode a database...
Vibecoders are the modern version of YouTube fake musicians:
https://youtu.be/FRoXKvD19Po
https://youtu.be/s_mN0DpOttI
I guess MongoDB was also vibe-coded back in 2013: https://aphyr.com/posts/284-jepsen-mongodb
MongoDB had consistency issues when running as a distributed system, that's expected if you read Jepsen's reviews. Meanwhile, ParticleDB had bugs like this when running as a single node:
MongoDB could lose your data on a single instance after acknowledged write. And corrupt the entire database in case of server crash.
Yes, this is worse, but in both case the root cause was human carelessness, not AI.
The old joke was that MongoDB copied MySQL's BLACKHOLE very well.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available. It is unclear what properties might be proven, or whether they described ParticleDB as built.
There are so many people using TLA+ and Lean via LLM without realizing that the proof might not even prove anything about your code
[dead]
It’s wild how bad it is. Not surprising for a vibecoded database, but why did they hire Kyle for a test, instead of first doing ANY testing themselves?
My best guess is their board/investors forced them to.
In this theory, without evidence just like their db claims, they were looking for external evidence to shut it all down or force replacement of key personnel.. And Aphyr understood the assignment.
I'd assume that no-one who works there really knows anything about databases; they may genuinely have thought that it worked. A lot of super-AI-pilled people seem to be deep into this "expertise doesn't matter any more" belief system.
No doubt because the LLM said it was ready to pass Jepsen.
^^ this is funny, but I think that's likely how it has happened.
And no doubt they wanted a headline to help boost the Anthropic IPO (I'm kidding but knowing the crypto crowd and the overlap with the AI crowd, who knows)
it is surprising even for vibe coded database.
Imagine bringing in any outside testing before your WHERE implementation works.
[flagged]
> We present 96 issues in ParticleDB, ranging from the pedestrian (crashes, deadlocks, data loss, etc.) to the piquant (partially-invisible rows which could not be deleted, WHERE 4 = 5, dropped tables re-emerging in distorted forms, confused columns, strings turning into integers, and so on).
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
The discussion section of the report is very damning.
[dead]
The AI boosters keep complaining that their work isn't evaluated on technical merit. Thank you for providing strong evidence that "the abstraction layer just moved up" mindset and vibing all your code, infra and culture does not lead to high quality software.
https://github.com/particledbai/jepsen/issues/18 is utterly humiliating. I can’t believe they kept going after getting that report. I would die of shame.
(Edit for clarity: this is quoted from the linked article)
Good on them for being good sports I guess, but after reading this I would never touch a piece of software built by anyone involved with this company. The sheer hubris of their claims combined with the specific reality of their broken garbage software (with the contradicted marketing claims and hallucinated documentation) is literally breathtaking.It must have been darkly amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so-slowly start to realize their software quality is not quite what their glazing agents had been telling them and that what they have on their hands is basically sub-mongo-quality slop. What a world we live in, huh?
I won't push back on it being garbage, but I would suggest that if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork, e.g. [1]
I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL. Your example queries remind me of that.
[1] https://medium.com/dailyjs/the-why-behind-the-wat-an-explana...
NULL means what it means and is conceptually foundational to SQL. Comparing it to anything returns UNKNOWN, the “third value” besides true and false. And WHERE needs a TRUE predicate. So your implied fault that they “have no problem“ really is just a matter of you not understanding the tool you are complaining about.
To match against the NULLness of a value you must use special purpose operators. In this case
Are we talking Oracle null or ansi null? I'm just being sarcastic and I don't actually remember all the fed up s** Oracle did
Oracle's weird null behaviour is that there is no empty string, so ('' IS NULL) is a true statement in Oracle.
That is weird, good to know!
Hi apologist. How was the woodwork? Your exact comment was why I wrote mine.
Cool, definitely_not_a_column doesn't exist, so therefore that whole WHERE clause doesn't exist either, so the real query is SELECT FROM t. We're comparing 4 and 5. Precedent? What happens when you treat -1 as a bool? The following languages agree that -1 is true: C, C++, JS, Python, PHP, Ruby, Perl, Lua, R.So it's SELECT FROM t WHERE true;
Now, let's get busy blaming Jepsen for using the tool wrong.
> therefore that whole WHERE clause doesn't exist either
No, therefore the condition is unsatisfiable and equivalent to WHERE FALSE. And that's letting you ignore the SQL spec, which says the query is invalid and should return "Column not found" because the column name cannot be resolved.
> We're comparing 4 and 5. Precedent?
I really hope you're trolling, for your own sake.
The explanation is more damning than the devex.
The condition is unsatisfiable, and the engine went ahead and gave a 0-warning result?
Are you claiming that 4 = 5?
I'm claiming that
1) It would be no more fucked up than anything else that has been grandfathered into our tools.
2) If it already had mass adoption in the market, people would jump onto this thread to explain to me that I just don't understand the brilliant semantics behind language of why 4 = 5.
I spent more time than I’d like to admit trying to understand your comments here. I’m convinced you are a very odd person with a confused mind.
Brilliant semantics? No one said anything normative. You’re arguing with yourself on your own non-sequitur. SQL is SQL. It’s extremely flawed but who freaking cares, that’s why ParticleDB shouldn’t have tried to implement SQL let alone full postgres compatibility when they clearly are not equipped to do so! Your comments about how strange and non-intuitive SQL are true but irrelevant.
Apologist for what?
> I won't push back on it being garbage, but I would suggest that if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork
No. No-one would ever accept a database with _this_ level of brokenness. Like, this makes early Mongo and noughties-era mysql look _amazing_.
> I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL.
You may not like it, but that is entirely deliberate and internally consistent. Blame the people who came up with SQL in the 70s; far too late to change it now.
I am rather receptive to the argument that SQL is a fractal of bad design. But complaining that SQL engines implement well known SQL semantics according to spec is silly.
"SQL engines implement well known SQL semantics"
That's why I preplied:
> if enough people did it that way it would be the new normal, and apologists would fall out of the woodwork
If 4=5 made it into the spec, would it be no longer up for criticism?
it's unfortunate that you are getting downvoted for having an actual conversation rather than just piling on these guys. but there are also a lot of places where people have said "everyone is using LLMs" and also "this thing about the job being requirements is a myth, because lots of requirements are ridiculous" has also been said, so i don't know. what do people want? a lot of these bullshit startups to fail, which is funny, but it's not good.
They’re really not having an actual conversation. You’re mistaking unnecessary argumentativeness for something more than it is.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof.
Perhaps they meant formerly, probably just a speech to text problem
TLA+ doesnt prove the code. if they never read the proof, then how do we k ow that it proved the right things?
And you still have to make sure all the real constraints are in. I had to fix a proven correct distributed algorithm that just happened to believe updates were done atomically and with no latency.
lol, reminds me of the common misreading "provably safe" / "probably safe"
I am wondering how this was allowed to be tested from a corporate strategy point of view of the company Orkes.
This is such a huge reputational hit, seems like a terrible outcome for a Series B company funded by big name VCs
What does this say about their main product?
> how this was allowed to be tested
I can see it happening if they genuinely had no idea of how defective / non-existent their internal QA & testing processes were.
Exactly, this is the takeaway. The agents are running everything, the humans have no idea what is going on.
You know how some people think that LLMs are magic? I assume that's more or less what happened here; the database was a result of prompting the finest LLMs, so how could it _not_ be perfect?
Yeah, wouldn't touch anything this company makes with a ten foot pole.
100% AI psychosis. They are all high on their own supply.
If anyone had any fucking clue they never would have engaged with Jepsen with the software in this state.
This company and everyone involved are going to have to legally change their names to have any shot at future trust.
It is a database for AGI bro. Sounds like you are not psychologically ready for the AI revolution in the age of AI.
> Strings sometimes turned into eight-bit integers (#37)
FINALLY we know what happened with eMudhra's keys!
(Sorry. Really couldn't resist.)
They could have prevented the entire debacle by running it in a four loop.
Shipping a bug like "WHERE 4 = 5" returning rows says it all to me
Maybe you don't know too much about engineering, but things happen, bugs are possible. Rome was not built in a day. All systems have bugs. The source being closed, with no QA, what do you expect. Maybe you don't remember Postgres in the early 2000s.
I remember Postgres in the early 2000s, and it didn't have anywhere near this level of problems. It had its issues, especially in terms of performance, and there were of course occasional correctness bugs that tended to turn up when you were doing something unusual. But you could certainly rely on basic SQL operations to work correctly.
> Maybe you don't remember Postgres in the early 2000s
Postgres was already mature by the early 2000s - it was started in 1986 by a veteran who had written Postgres' spiritual predecessor in the 70's (Ingress). So no, Postgres didn't have amateur-hour data loss bugs in the early 2000s. What Postgres was missing out of the box were some features, like replication.
> So no, Postgres didn't have amateur-hour data loss bugs in the early 2000s.
Well not until you'd been running it for a while and it ran out of tuples at least.
My argument wasn't that Pg was bug-free... In any case, Postgres was always superior to MySQL, whose default engine would silently truncate your data.
The MyISAM days were the wild west.
> Maybe you don't remember Postgres in the early 2000s.
Not even the grungiest version of MySQL 3.23 running on some shared hosting provider in the early 2000s would behave as immediately illogically as the database under test here.
Someone with your username should know this.
Created 5 hours ago, prob astroturfing.
I remember Postgres in the early 2000s. And early Mongo. And many others.
And bugs like “WHERE 4 = 5”, or selecting from nonexistent columns, or selects permanently modifying table behavior, or any of a dozen others that Jepsen found … didn’t happen in released databases. Even young ones.
Prior to the age of LLMs, that level of bugginess wasn’t “everything has growing pains”. That’s the kind of thing previously associated with poor undergraduate CS homework or “the CEO’s kid wrote our database” corporate horrorshows.
> with no QA
That’s not immaturity. That is a choice. A “the car we are selling does not have airbags, seat belts, or brakes” kind of choice.
> Maybe you don't know too much about engineering
Save it. You have no idea what you are talking about.
> Maybe you don't remember Postgres in the early 2000s
The ParticleDB devs definitely don't
@dang I think HN should revoke and reserve this username.
Edit: I know @dang does nothing but maybe make it more likely he sees it.
You know what absolutely does work and gets mod eyeballs?
writing to hn@ycombinator.com from your account associated email address.
Imagine asking for a review before checking with the brutally powerful agents we have today all the low hanging fruits. And, at the point we are, you can likely go at incredible levels of automated bugs seeking and do the whole test in house.
My understanding is that Orkes did, in fact, ask AI agents to test ParticleDB in-house prior to our collaboration.
It is months that I find subtle bugs far more complicated than the most immediately findable issues in the report. Together with the rest of the world. So there must be something wrong with their AI usage.
Right, exactly. All those disclaimers in front of various AI applications have made people forget that in fact, AI cannot fail, it can only be failed.
I do not know in what order Kyle et al. wrote the report, but as I progressed through different parts I think I could detect instances of an increasingly fatigued/disheartened/unhinged tone--without compromising the professionalism we expect from Jepsen analyses, of course. These moments of humanity brought me joy.
More cynically, I suspect that many low-quality database projects succeed or fail based more on marketing and hype than robustness. A dishonest product could simply take the line "engineers at Orkes were able to produce a highly sophisticated database system" from the report out of context, quote it on their homepage attributed to aphyr/Kyle/Jepsen, and reap short-term rewards and popularity. I hope I'm wrong.
I think linking to TikTok videos saying “vibes? vibes!” in some of the links was a bit below the belt, but understand it takes restraint to maintain NPOV
I'd guess that all the restraint in this case has been spent on staying professional throughout writing all the individual bug descriptions. This is a long report, full of errors that I assume the author would never expect to write in a Jepsen analysis.
Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
The company says they did. From the post
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
They probably asked an AI to run the tests against their DB. And the AI said "yeah I ran it, all green!" (except with a lot more words and perfect grammar, and maybe a few section headings and emoji.)
Seems to me that “we ran the Jepsen test” is just something an LLM thought would sound good on a DB’s website.
Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.
Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.
The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.
Unhinged announcement blog: https://orkes.io/blog/particledb-database-built-for-ai-agent...
tigerbeetle much different result
https://jepsen.io/analyses/tigerbeetle-0.16.11
https://news.ycombinator.com/item?id=44199592
A long time ago I worked with Aphyr to evaluate a database with Jepsen. He was extremely professional, and he found a few real issues which we resolved extremely urgently. We took things seriously and wrote a significant amount of testing ourselves, and by the time our engagement began, we were pretty confident. Overall it was a great experience and I highly recommend his services.
By comparison, this report paints an absolute clown show. The amount of anomalies is perhaps understandable (a single architectural flaw can have a lot of impact) but the fact that they found so many disparate issues is telling.
The fact that they then said they addressed the issues, but didn't, is about the biggest red flag imaginable. I would never let any client use this software under any circumstances.
My colleague analyzed their ClickBench submission on May 3rd and concluded that:
- "This is probably the most sophisticated spam attempt in ClickBench so far"
- "Unless someone tells me otherwise, I assume ParticleDB is fake."
Aphyr does great work, Jepsen always kicks it out of the park :)
It’s really great to see more of these @Aphyr, I really enjoy them and learn something from each. Thank you!
Slop DB written in Rust. They probably changed the landing page after the bad results, you can still see the old one: https://web.archive.org/web/20260518203012/https://particled...
"Written in Rust · Formally Verified · PostgreSQL Compatible"
To everyone pointing out that artificial intelligence is to blame here, you may want to read his takedown of MongoDB back in 2013: https://aphyr.com/posts/284-jepsen-mongodb
Is there any concrete evidence they vibe coded it? It seems 100% self-evident, but have we actually ruled out human hubris and gross incompetence?
> have we actually ruled out human hubris and gross incompetence
it takes a lot of of human hubris and incompetence to think you can herd AI agents to squirt out a better version of a multi-decade project in weeks, without a single human in the loop. I can't imagine the levels of hubris in prompting "Make a faster Postgres in rust. Make no mistakes" and immediately uploading the result for Jepsen testing without bothering to manually kick the tires a bit to see if your DB is fit for purpose.
I see a direct line between this attitude and the Proton CEO deriding "loser thinking" in a subordinate that dares suggest the AI output have some oversight before going to prod; there exist people who are so AI-pilled they get lost in the sauce.
Vibe coded or not I think human hubris and incompetence played a big role here.
Great to implement a DB just like human memory, unstable, and untrustworthy.
Every database vendor should pay to get their software Jepsen-tested.
I am not sure I would want my database tested by a vendor who feels it their duty to provide color commentary on my software development methodology in a public report, no matter how rigorous their evaluation.
> vendor who feels it their duty to provide color commentary
Honestly? At some point there's a fiduciary-like obligation for it. What ParticleDB delivered is so far past good or bad, it's just straight ugly and then some. It's so poor that, if Jepsen couched it in any way, they'd be implicitly misleading their readers.
The principal values of things like the Jepsen reports is to the community of potential users/purchasers.
I agree with GP, that vendors should be somehow required or very strongly encouraged to participate in such audits. Not because they want to (as you say, most wouldn't like their dirty laundry aired thus), but because consumers deserve to know. U.S. government software groups don't like FISMA/RMF audits either, but they have to do them anyway.
I am of a contrary opinion. At some level of developer insolence, such commentary is valuable - especially if it needs to mention the core basics and foundations that keep the software world running, such as running existing Postgres test suites for a supposedly Postgres-compatible database. In the increasingly sloppified world, even such basics are being eroded - this being a proof.
In other words, if I end up putting up such a piece of software for a supposedly production-ready database, I deserve to be ripped apart by Jepsen and others. Somebody has to assert software quality; if it's not asserted internally, it must be asserted externally.
In yet another words, software development is a trust-based business. If you break it, such as by outright lying to people, then you deal with the consequences.
Fatality
Aphyr (and/or his team) is the best technical writer in the biz. These Jepsen reports are always so clearly explained. This one is fun because the tone occasionally verges on disgust.
I've never heard of this system. Came out of nowhere and closed source it seems. Could not find much analysis on it at all which is not the best sign.
Damn aphyr if there were ever a time to write one of your old school meme filled reports it would have been this one lol
Well done. Nice D&G riffing in the endnotes too
Is/was ParticleDB open source? There is a Github link on their website, however the only public repo in their org seems to be the Golang client library.
https://github.com/orgs/particledb-ai/repositories
SDKs are MIT license CLI tools and Studio will be also be MIT License
Server is not open source for now but free for developers
“No.”
Afraid to show the slop?
No, they’re aware they don’t possess copyright on the source code because it has not one scintilla of human effort, so keeping it secret is the only legal way to protect it! :)
How did the test for 6 = 7 go?
Dude please set up an RSS feed I've been reading your jepsen posts since like 2014 and haven't realized how much i missed.
You're in luck! https://jepsen.io/blog has Atom and RSS feeds. :-)
Oh, jeez, sorry! There's my Sunday afternoon project, I guess. :-)
Thank you it's working now!!
Clicking on either of the feeds takes me to a page that just returns an error.
e.g. https://jepsen.io/blog.rss returns a text/plain respond containing only "Nothing to see here; move along.".
Those feeds appear empty, though. Although I could be running into a race condition, since it's the Jepsen site.
I’m looking forward to the Jepsen review of pgrust, the other vibecoded DB. Plenty of hype and slop, with a side of fake
Now do ArcadeDB, please. I know it's full of bugs. They fucked up DFS vs BFS and it was in the codebase since they copy-pasted the code from OrientDB.
From ArcadeDB dev team here.
We have implemented the Jepsen tests by our own: https://github.com/arcadeData/arcadedb-jepsen/
Not all of them are green, we are working to pass them all.
DFS VS BFS: what do you mean? Both are documented: https://docs.arcadedb.com/arcadedb/reference/sql/sql-travers... . And are supported under Cypher as well: https://docs.arcadedb.com/arcadedb/reference/graph-algorithm...
the only bug related to this, and coming from OrientDB (we all were part of the OrientDB team) was solved in 26.7.2 (July 2026): https://github.com/ArcadeData/arcadedb/issues/5159, coming from https://github.com/ArcadeData/arcadedb/pull/5135.
As any other project out there, we have bugs, but we solve them in public. We have more that 30k unit tests in place, ha-integration tests, ha-resilience tests, chaos tests and more. Coverage is almost at 80% (https://app.codecov.io/github/ArcadeData/arcadedb) Are all the tests green? No, we have some flaky tests and we do our best to fix them form time to time.
If something isn't working on your side, please file an issue
glorious!
No one’s ever using it after this. Sounds like an unnecessary pile of shite.
I wonder what the internal testing pipeline looked like before this. A codebase with that many fundamental issues suggests either no integration tests were written or they were all mocked out to pass green. It seems strange that a team would market a system as formally proven without actually running the workload through even basic consistency checks themselves first.
That part is the weird bit to me. If the existing harness breaks right away, that's the signal to stop and fix things, not to book an outside audit. Running the suite in CI on every build would have caught a lot of this.
> Running the suite in CI on every build would have caught a lot of this
yeah, provided:
Presumably a few of these things weren't being done -- I'd bet 1 & 2 were skipped or implemented so poorly that they were worse than having no tests at all -- checking nothing but giving a false illusion that some kind of testing had happened.Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.
They forgot the “don’t make any mistakes” part of the prompt.
-https://github.com/jepsen-io/particledb is currently 404 (probably still private?)-
EDIT: It's up now
Sorry about that--yes, it's still private. Hoping to have that resolved shortly.
Looks like it's up now. I'll put that link in the toptext as well. Thanks!
[dead]
[flagged]
In that case Ant/OAI should give Aphyr a blank check and infinite access to tokens to build the Mother-of-all-Databases. Anthropic has a few days before IPO day, they better get to it!
Congrats to the particledb team for getting this far and thanks to @aphyr for the analysis.
It takes gumption to build a new storage engine. I remember when etcd[0] first made it's way into @aphyr's cross-hairs. We took all of the input very seriously because we knew that a data store which is untrustworthy is useless.
Resolving these issues likely won't be easy, but if you can get to the point where you walk away from Jepsen unscathed the trust in your software will be greatly amplified.
0: https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
Finding issues is normal. But the bold claims they had on their website is what makes this a bust.
Right, when Rachael Gunn (Raygun) embarrassed Australia and the whole of breakdancing on the olympic stage, the nice thing to say would have been "congrats on getting this far, fixing the issues won't be easy, etc..." but what really should be said something like "I'm not sure how it got to this point, but everything about this is wrong, and you should probably focus on something else".
Reads like investor cope. And the OG etcd was a disaster that needed better engineers to completely rewrite. The surviving project is an entirely different lineage.