Hi! I'm the creator of pgrust. Let me explain what happened.
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
Agreed. Though we still don't have a reason why they rushed a custom preview to a rigorous benchmark competition. Feels like too much hype, and now damage control.
I got it slightly wrong. It was code that you could run with an environment variable: PGRUST_GRAVITY_IRPROBE. It still wasn't intended to be part of the actual binary.
This comment chain is hilarious. Did you ask Claude or another LLM what that file was, then ask it the question from OP comment (how it was included in binary), and it admitted it was wrong?
Because this is such a common pattern when working with LLMs, and this looks exactly like it.
No. I remember looking into this before, back when we were originally accused of cheating. I knew the code was dead and unused, but I got the reason wrong and had to double-check.
Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
This absolutely does not pass any smell test, by your own admission you included the test files "by accident", could only barely pass the minimum of SQL for the benchmarks, included environment variables for "internal benchmarking" and a preview version you cannot immediately release because it might contain sensitive data? What kind of sensitive data are you putting in a Postgres slop fork in such a way that you cannot immediately release it, besides obvious admissions of cheating? This all just reeks.
> This absolutely does not pass any smell test, by your own admission you included the test files "by accident"
If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
I mean, if the now-released open source variant performs roughly the same, and has no weird behavior in it… the matter is resolved, no?
They’ve already submitted the best possible corrective step for any sus behavior they previously did.
It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
Honestly, if the database is not able to be run with a variety of queries other than the one used by ClickBench then don't submit it to the benchmark. Every other database in their list can run anything you throw at it. Hiding behind messaging that it was an early 0.4 build that can't do everything is ... weird! Just don't submit it to ClickBench if it is not usable by the world yet.
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
Yeah, it was a mistake to submit it as early as we did. I was very excited when we got the early version working and overeagerly submitted it to ClickBench
0.4-preview was an early version of 0.4 that still had some cruft like the feature flag you pointed out. We removed the cruft for the release of 0.4-beta
Why don't you open source 0.4-preview too, to make your claims verifiable. The longer you take to release it (or choose not to release it at all) the more it'll look like a cover-up. If the binary is (made) public, it's going to get the AI-assisted decompilation treatment anyway.
The new general version is actually faster than the version accused of cheating.
The architecture of the query engine is:
SQL -> IR -> Assembly
-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.
If it delivers on being faster than the version accused of cheating with reasonable cache settings and no SQL feature limitations, I would say that resolves the accusations, regardless of the source code of -preview.
The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.
> For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview.
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
> This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview
That statement provided context and was not part of the defense. The point was that -preview didn't have the issues described, except for limited SQL support.
If pgrust were backed by a company that advertised these benchmark results, you would have enough to claim fraud in a civil lawsuit. Literally snaking oneself out of accountability after being caught is dishonorable. Children do this, not adults. Seeing adults act in this manner is pathetic.
This is an accurate description of one's bad behavior.
If you believe these word choices are inaccurate, we can discuss, but mischaracterizing the reply as "aggressive" is just bad faith (and against the rules).
Calling your reply aggressive isn't against the rules, bad faith or not (multiple people on HN clearly think it is aggressive). What is in the Guidelines, however:
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
In the past, we've done that because we've tended to do large refactors, many of which we've backed out of. For that reason, we've kept our own internal development version, which is ahead of the public version. Then when whatever we're developing is ready for public use, then we open source it
Why not keep that internal version as a dev branch separate from main? I can see the appeal of keeping things out of master/main for a while, but not why you would keep that part of development hidden.
But most importantly, why let out the binary and not the source? presumably if something is not ready for public use, the artifact being out is worst of both worlds - unready code is out and the public can't check what it does.
Not trying to argue mind you, but if this was not a deliberate attempt to hide the binary's behavior I think I'm missing the alternative upsides that justify that flow.
The dev process carries over from when we constantly rewrote the codebase from scratch every few weeks[0]. We ran many experiments, most of which we threw away. A small number became part of "pgrust". It didn't seem worth it to maintain public branches with all of our development effort when we knew 80% of it was going to be thrown away
It makes sense. Just because one can do a thing doesn’t mean they must want to do it.
It’s like everyone who squash merges for a clean timeline. You want you dev process to look prestine and communicate what changes were kept, not every experiment that failed.
That model of development is what led to Eric Raymond’s seminal The Cathedral and the Bazaar essay.
“Cathedral” refers to development behind closed doors, where you just get a release and a source dump to go with it.
“Bazaar” refers to development done in the open, where you have constant access to the bleeding edge.
At the time of the essay, GCC and Emacs were ESR’s examples of Cathedral-style development, and Linux was the premiere example of the Bazaar style.
The Cathedral style has become a lot less common over the years, but it’s still how e.g. SQLite operates, I think? E.g. Google definitely operates like that with AOSP, but that's arguably more a fruit of them being maliciously compliant with their Open Source promises.
Some projects do, for example the Lua programming language does their work in private, between releases.
Other projects do it less successfully, making grandiose claims and later releasing things in a partially working state, for example the V-programming language.
It's not super-common, but just those two examples off the top of my head show it's not entirely unheard of.
Not normal. This a new trend of people who use too much Ai, or maybe more specifically the Claw/Ralph loops where you end up with a few commits touching 1000s of files. For about the last year, many Show HN to github projects are often just a single commit
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
Let any who read this be aware: You should be immediately suspicious of AI code that is much much faster than established code bases. AIs are even better at benchmarkmaxxing than humans are, and humans were already pretty good.
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
The project author seems to have prompted Claude to add a test to catch Claude cheating, but will Claude change this test as a workaround in a future commit, no one knows because no one is reading the project code...
I am getting increasing disillusioned by the number of peers I am seeing completely hand over their thinking to a tool. Agents are great, but if you can't explain to me how your PR changes a system and why design choices were made you are no longer an engineer.
Agreed. Used to be prevalent in the frontend world (now I think it's the only thing that exists seemingly), where programmers would copy-paste stuff from Stack Overflow until things worked, and when you asked "So why did this actually fix the issue?" they usually said "I'm not sure, but it works now, that's what matters". This seems to be standard practice across the entire stack now.
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
This seems more a product of the incentives we have in society, namely seeing the rich and wealthy on social media and then wanting that quick and easy. So many of those cases are not people with passion for the problem.
> Good things take real effort, bring back the TLC!
Ain't no one willing to reward the real effort any more. I take pride in not using LLMs beyond the capability of IntelliJ's advanced autocomplete, but then again, I only code for fun these days given I fled IT for electrical engineering.
What we don't want to do is mirror the partisan political environment in the Ai yay, nay, or how. We need to be able to talk to each other in understanding and a shared reality of facts. Too much hyperbole right now, starting from the top of the big US labs. They need to step down a few ego and sycophancy pegs
Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.
Either way you don't want to be platforming the product because neither shows any degree of responsibility.
That comment doesn't evidence cheating, it reiterates that they submitted the result prematurely which was already admitted in the comment that your link was responding to.
EDIT: removed a lament that people don’t read comments they respond to, in case this violates the site rules.
So, the charge was that they submitted a build specifically tuned for clickbench
The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else...
Doesn't feel like much of a defense to me. At best it's benchmaxxing if not clear outright cheating
> The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else.
ClickBench tests a very narrow set of SQL: scans, aggregations, sorts, and filters. It ends up being a great starting point for building a query engine before layering in more functionality
> The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else...
According to both the ClickHouse founder who was levying the accusations and the pgrust creator, the submission worked for many other queries it just didn’t have full SQL support.
Whenever people benchmark something my default assumption is that they want to know how well it performs, but I’m not the author and can’t speak to their intentions.
I think it is reasonable, if apparently incorrect, for people to assume that your account is a throwaway account, given its name. I'm surprised that you're surprised by this.
Account names are just unique strings within a set of allowed characters. I could make my handle “TheKingOfEngland”, but it wouldn’t actually make me English royalty. I thought in a community of programmers this would be self-evident, but alas not.
I’m not sure if they’re equally reasonable, but inferring something from a username is at least as foolish as making inferences based on account age. The critical distinction, however, is that my account age remark was tongue in cheek while you joined the fray to defend drawing conclusions from a username.
> inferring something from a username is at least as foolish as making inferences based on account age.
That's just not so, friend.
Again, when 99.9% of usernames with the form throwaway+random number string are throwaway accounts, it's reasonable to assume any account fitting the pattern is such an account, because odds are astronomically high that it is. This has to do with something called convention, and I'd suggest reading more on that topic and how it can affect inferences.
Whereas, there is no indicator about how any account may tie to the experience of the person posting behind the account. It's as useless a metric as account name itself for trying to gauge that, and honestly, that should be obvious.
> is that my account age remark was tongue in cheek while you joined the fray to defend drawing conclusions from a username.
You put the emoticon to indicate a perhaps tongue in cheek response, bur the rest of the text and conversation didn't really support that, nor do the continuing replies.
I merely corrected you when you tried to say it wasn't reasonable for people to assume an account named such as yours is a throwaway, because it very, very much is.
> Again, when 99.9% of usernames with the form throwaway+random number string are throwaway accounts
Your entire argument is predicated on this completely fabricated statistic.
> You put the emoticon to indicate a perhaps tongue in cheek response, bur the rest of the text and conversation didn't really support that, nor do the continuing replies.
I’m the authority on my own emotional state, but also as a native English speaker I can assure you that our dialogue thus far comports perfectly with someone who was speaking tongue in cheek, but who also doesn’t mind watching people try to dig themselves out of holes they created for no discernible benefit :shrug:
I'm not, I think you're ego is preventing you from admitting you said something silly, however.
> Your entire argument is predicated on this completely fabricated statistic.
It's not fabricated in the least.
> I’m the authority on my own emotional state
Sure, that doesn't mean what you claim is honest however. Plenty of people would claim to be joking when they were anything but, for all sorts of reasons.
> as a native English speaker I can assure you that our dialogue thus far comports perfectly with someone who was speaking tongue in cheek
Native English speaker here also, and I can assure you that your dialogue thus far also comports perfectly with someone who was not speaking tongue in cheek, but asserted they were so they could hopefully put an end to a conversation that was started by calling them out for something silly.
> but who also doesn’t mind watching people try to dig themselves out of holes they created for no discernible benefit
Funny, that's what I'm watching here. It's a hobby of mine to recognize people who can't admit when they are wrong and need to have the last word, taking it to an extreme level. There isn't much more to be gained from this conversation since you refuse to be reasonable and acknowledge longstanding convention regarding the naming of throwaway accounts, but I am curious how much longer you will drag it out.
You joke but when benchmarking with ai this is important. When you tell it optimize X, it will do so at any cost, including cheezing benchmarks. So you have to make sure you tell it what it can’t optimize along with what it can.
Tbf, the author (prompter?) went on Postgres.fm [0] recently to discuss it, and while it didn’t make me entirely confident in it, it did at least seem like it was several steps above the average vibe project.
A lot of podcasters don't have the expertise to evaluate Ai generated/assisted output. I see this much more outside of the development community. Algorithms have warped the incentives and people have stopped doing basic due diligence.
To be fair to the postgres.fm guys (who I do like and appreciate), the pgrust code was not available a the time of recording.
As much as I understand the urge to step in when somebody is wrong on the internet ... The R in PR is for "request", isn't it? A merged pull request is still a request (which was satisfied/merged).
If I ask you a question and you answer it, and then I show you a link to a document containing my question, then what somebody finds when following that link is still a (now-answered) question, isn't it?
It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.
The facts around the cheating are disputed. It's possible the accuser was mistaken about what was happening. To me the main evidence would be whether the fully open source version matches the alleged cheating performance or not.
We were the agents for the models all along, with our highly adaptive tools: versatile sausagey appendages and flapping fleshy orifices for interacting with the world
So I spent some time poking at the released 0.4-beta code. Controversy aside, it is interesting to see an open source system that beats ClickHouse on ClickBench - 4 out of the 5 systems ahead of CH (Umbra, elosdb, Hyper, CedarDB) are all closed-source, while the only open one is pivotlake.
What makes pgrust fast? There's no one single reason, but a combination of techniques like dictionary encoding [1], zone maps [2], late materialization [3], vectorization [4], and a gazillion others. It's not unlike all the countless optimization passes in a compiler.
None of the optimizations are groundbreaking - in fact, I could find a counterpart of every optimization I checked in ClickHouse as well! So why is CH still slower?
I believe the difference is in when they decide to apply each optimization. Unlike compiler optimizations which are usually beneficial, DB optimizations are highly data-dependent, and the query planner needs to turn them on an off based on data statistics. For example, CH is more conservative in applying dictionary encoding, and it basically doesn't get used at all for ClickBench.
Turning off dictionary-related optimizations in pgrust immediately made it 5x slower (from 3.8s to 20.4s on my machine), while ClickHouse takes a total of 15.0s.
This points to one important nuance in benchmarking - when all you target is one single benchmark, you can be more aggressive during query planning, as plans that work well for one query probably also work well for others, due to characteristics shared among queries/datasets. But when you build a system that needs to support millions of users, you have to be conservative with optimizations.
Sounds kind of like Claude was "cheating" and the humans are engaged in eyes-free coding. Diff so big github refuses to render. I suspect they are reaching the limits of what this new vibe "paradigm" of programming can do. If you haven't pushed a vibe project far, eventually progress becomes near impossible because the slop has grown and the conditions are overly complex, locally focused. Ai is not good at writing reusable software (on their own).
The prior (2nd) commit with ~1700 files at least shows the file tree and such a warning, the latest (3rd) commit is something so big, GitHub will not even do that.
Yup. These are people who have gone full Claude. The dopamine seeking is at previously unseen levels. they think this pr is some factual item that means something so they post it on HN as click bait. But no it only means something to a fulltime ai slopper
If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
I've noticed it with a lot of random stuff i've been trying
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
You can't compare optimizations done on audio encoding/decoding (often hand written assembly) to something like PostgreSQL which has many moving/inefficient parts.
The optimizations on PostgreSQL are real if done correctly, it's just a very big project.
My point was that while Claude _is_ legitimately very good at optimising. It will also write its own benchmarks to test its optimisations which are frequently stacked in its favour. You have to be very careful that reported speedups are actually real
If I saw that test in a PR I would not only reject the PR and ask a lot of questions but someone would likely take a forced vacation to deslop their brain.
> If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
Depends on who wrote the original code. With postgres its sus, but if it's another AI...I wouldn't be surprised. I also saw code writtem by humans that is a result of "I can't figure out how to make this safe and concurrent, so imma just place giant mutex here", and it like 100 lines change to fix that.
These invectives are verging on slurs at this point. You don’t know anything about the chemical seeking disposition of the author who is actually here in the comments, so why don’t we just stick to facts instead of loading posts with hyperbole and name calling? All it’s does is invite name calling and hyperbole back. You can express your frustration in ways more conducive to productive discourse.
The post is garbage. There shouldn't even be discourse other then "this is what slop looks like over the course of a few months". So that's what I offered. The best anyone can do
It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
Anecdote: I have seen it cheat / bend the rules with my own eyes.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
My own anecdote: I have seen frontier models get stuck repeating the word "the" until it fills up the context. I really don't care to argue the capabilities of LLMs or their somehow simultaneous lack thereof; I'll just reiterate that ultimately the operator chose to ship a cheat.
I believe them. I had Gemini-3.1 Pro do essentially this seven months ago - specifically a loop of writing "Bye! Done!" repeatedly until manually stopped. Presumably some sort of pathological behaviour where it wasn't ranking the stop token highly when it should have been... but damned if I know why... it was an innocent prompt (and it had done the thing I asked it to before failing to stop).
It's really uncommon now, but it's still possible. I hit it two days ago with deepseek 4.1 flash, and I've had coworkers report that happening with Gemini.
"loss of control" is a phrase being used for a reason, the agents do things we do not want them to, even in the context of instructions to the contrary. The current training regime has incentivized "complete the task any and every way you can" - the bot judgement is very poor across the board
Even before the ai age, people published inaccurate/misleading benchmarks all the time. Sometimes those were actual cheating, and other times an honest mistake. What matters is that the author acknowledges the issue and attempts to correct the mistake, which clearly seems to have happened here: https://github.com/ClickHouse/ClickBench/pull/2751
Most of this thread doesn't seem to acknowledge that, and feels more like anti-ai psychosis using this incident as an anecdote to justify the pre-existing hatred of ai-generated projects.
I suspect the author of that PR submitted it and attempted to correct the mistake because he got caught. Forgiveness because he reneged after getting caught encourages the behavior. The project, nor author should not be trusted; otherwise is naive.
"suspect" meaning speculate, as far as I can tell. not really anything to argue with on that front. I never denied there was a mistake. Cheating implies intent, which no one here can prove, so seems useless to debate. Open source development is awesome in that it keeps people to a high bar, let's just celebrate that the process clearly worked here, and not muck about in speculation and judgement of others intentions that we have essentially no evidence for.
They claim they were not cheating. The open-source version without the alleged cheating devices is as fast if not faster than the supposedly cheated results, which is a bit hard to explain as a hasty attempt to cover up cheating.
Its like a lot of people really want pgrust to fail.
There maybe a chance this was cheating or it may have been a misunderstanding. But now since there is an open source version, why is there still this strong sense of negativity.
I'm also not going to cheer on database projects that can't be 100% transparent and honest.
Who the hell is going to trust their data to a people can't even be honest on benchmarks. What do you think will happen if (or when) there's a data corruption bug. What evidence do you have to think these same people will behave any better?
They failed themselves, we're just watching it happen.
Do you not believe their explanation for the accusations? To me it seems like if the released version is hitting or beating the allegedly cheated benchmark numbers then it seems unlikely they were dishonest in the first place.
Hi! I'm the creator of pgrust. Let me explain what happened.
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
[0] https://github.com/malisper/pgrust/releases/tag/v0.4-beta
[1] https://pgrust.com/downloads/archive/2026-09-clickbench-snap...
[2] https://github.com/ClickHouse/ClickBench/blob/main/README.md
[3] https://clickhouse.com/docs/concepts/features/performance/ca...
[4] https://github.com/ClickHouse/ClickBench/pull/2751
Thanks for the explanation, can you release the source of 0.4-preview so that people can understand what was going on?
I'll see what I can do. I'll need to go through and make sure there isn't anything sensitive in the source
I would recommend releasing the full history so the broader community can verify and you regain trust with those you lost it with.
Such as?
Agreed. Though we still don't have a reason why they rushed a custom preview to a rigorous benchmark competition. Feels like too much hype, and now damage control.
What's the explanation for paths to q00.rs-q36.rs being in the binary?
It was included by mistake. It was code that was in our repo that wasn't explicitly marked as test code, so it got included in the binary.
How? Compilators eliminate unused code.
I got it slightly wrong. It was code that you could run with an environment variable: PGRUST_GRAVITY_IRPROBE. It still wasn't intended to be part of the actual binary.
This comment chain is hilarious. Did you ask Claude or another LLM what that file was, then ask it the question from OP comment (how it was included in binary), and it admitted it was wrong?
Because this is such a common pattern when working with LLMs, and this looks exactly like it.
No. I remember looking into this before, back when we were originally accused of cheating. I knew the code was dead and unused, but I got the reason wrong and had to double-check.
Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
Yeah, that was a mistake. We should have waited until it was more complete before submitting it.
We were just really excited about what we had built and how fast it was that we wanted to submit it.
This absolutely does not pass any smell test, by your own admission you included the test files "by accident", could only barely pass the minimum of SQL for the benchmarks, included environment variables for "internal benchmarking" and a preview version you cannot immediately release because it might contain sensitive data? What kind of sensitive data are you putting in a Postgres slop fork in such a way that you cannot immediately release it, besides obvious admissions of cheating? This all just reeks.
> This absolutely does not pass any smell test, by your own admission you included the test files "by accident"
If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
How about you give us the source code to that binary so we can see for ourselves?
https://github.com/malisper/pgrust
That's not the -preview version which was submitted
I mean, if the now-released open source variant performs roughly the same, and has no weird behavior in it… the matter is resolved, no?
They’ve already submitted the best possible corrective step for any sus behavior they previously did.
It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.
Exactly!
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
Honestly, if the database is not able to be run with a variety of queries other than the one used by ClickBench then don't submit it to the benchmark. Every other database in their list can run anything you throw at it. Hiding behind messaging that it was an early 0.4 build that can't do everything is ... weird! Just don't submit it to ClickBench if it is not usable by the world yet.
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
Yeah, it was a mistake to submit it as early as we did. I was very excited when we got the early version working and overeagerly submitted it to ClickBench
Your explanation makes the removal very reasonable. Also pointing out to a version after this is done surely does not help.
thanks for the explanation, let's first see if the binary matches.
Can you explain "PGRUST_GRAVITY_Q20_VARIANT"?
It's a feature flag we used for our own internal benchmarking. It doesn't get hit when you run SQL.
But it's not in the code you open sourced?
0.4-preview was an early version of 0.4 that still had some cruft like the feature flag you pointed out. We removed the cruft for the release of 0.4-beta
Why don't you open source 0.4-preview too, to make your claims verifiable. The longer you take to release it (or choose not to release it at all) the more it'll look like a cover-up. If the binary is (made) public, it's going to get the AI-assisted decompilation treatment anyway.
thanks for your explanation. I'd love to see a concise explanation of the difference in benchmark performance between -preview and -beta
that would help clear the air, if the full-sql version has similar performance to the partial-sql version
The new general version is actually faster than the version accused of cheating.
The architecture of the query engine is:
-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.If it delivers on being faster than the version accused of cheating with reasonable cache settings and no SQL feature limitations, I would say that resolves the accusations, regardless of the source code of -preview.
The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.
The preliminary ClickBench results for -beta are in[0]. The geomean across all runs came out as 7% faster than -preview[1]
[0] https://github.com/ClickHouse/ClickBench/commit/56535d29e825...
[1] https://github.com/ClickHouse/ClickBench/commit/89cc76e1e4e2...
thanks, I think this is the most important point in your defense
> For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview.
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
> This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview
That statement provided context and was not part of the defense. The point was that -preview didn't have the issues described, except for limited SQL support.
[flagged]
You are being downvoted (edit: and flagged) because you are disproportionately aggro which undermines your point.
If pgrust were backed by a company that advertised these benchmark results, you would have enough to claim fraud in a civil lawsuit. Literally snaking oneself out of accountability after being caught is dishonorable. Children do this, not adults. Seeing adults act in this manner is pathetic.
This is an accurate description of one's bad behavior.
If you believe these word choices are inaccurate, we can discuss, but mischaracterizing the reply as "aggressive" is just bad faith (and against the rules).
Calling your reply aggressive isn't against the rules, bad faith or not (multiple people on HN clearly think it is aggressive). What is in the Guidelines, however:
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
You're still being extremely aggressive and assuming very bad faith. None of what you said is demonstrably proven, for a start.
This idea of "here's the beta, I'll release the code in a few weeks" sounds counter intuitive to me. is it normal in open source? Any reason for it?
In the past, we've done that because we've tended to do large refactors, many of which we've backed out of. For that reason, we've kept our own internal development version, which is ahead of the public version. Then when whatever we're developing is ready for public use, then we open source it
I'm not sure I follow the reasoning here, sorry.
Why not keep that internal version as a dev branch separate from main? I can see the appeal of keeping things out of master/main for a while, but not why you would keep that part of development hidden.
But most importantly, why let out the binary and not the source? presumably if something is not ready for public use, the artifact being out is worst of both worlds - unready code is out and the public can't check what it does.
Not trying to argue mind you, but if this was not a deliberate attempt to hide the binary's behavior I think I'm missing the alternative upsides that justify that flow.
The dev process carries over from when we constantly rewrote the codebase from scratch every few weeks[0]. We ran many experiments, most of which we threw away. A small number became part of "pgrust". It didn't seem worth it to maintain public branches with all of our development effort when we knew 80% of it was going to be thrown away
[0] https://malisper.me/postgres-in-rust-regression-suite/
But you can make the branch public without committing to maintaining an experimental branch, this doesn't make sense.
It makes sense. Just because one can do a thing doesn’t mean they must want to do it.
It’s like everyone who squash merges for a clean timeline. You want you dev process to look prestine and communicate what changes were kept, not every experiment that failed.
Have you happen to have seen the 3 commits? They are over 1000 file diffs, that is not pristine or communicating changes the way devs do.
No, it's the old school cathedral "throw code over the wall" style commits that some companies were famous for.
If you've been accused of cheating it is a very different scenario.
That model of development is what led to Eric Raymond’s seminal The Cathedral and the Bazaar essay.
“Cathedral” refers to development behind closed doors, where you just get a release and a source dump to go with it.
“Bazaar” refers to development done in the open, where you have constant access to the bleeding edge.
At the time of the essay, GCC and Emacs were ESR’s examples of Cathedral-style development, and Linux was the premiere example of the Bazaar style.
The Cathedral style has become a lot less common over the years, but it’s still how e.g. SQLite operates, I think? E.g. Google definitely operates like that with AOSP, but that's arguably more a fruit of them being maliciously compliant with their Open Source promises.
Some projects do, for example the Lua programming language does their work in private, between releases.
Other projects do it less successfully, making grandiose claims and later releasing things in a partially working state, for example the V-programming language.
It's not super-common, but just those two examples off the top of my head show it's not entirely unheard of.
Not normal. This a new trend of people who use too much Ai, or maybe more specifically the Claw/Ralph loops where you end up with a few commits touching 1000s of files. For about the last year, many Show HN to github projects are often just a single commit
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
Let any who read this be aware: You should be immediately suspicious of AI code that is much much faster than established code bases. AIs are even better at benchmarkmaxxing than humans are, and humans were already pretty good.
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
The project author seems to have prompted Claude to add a test to catch Claude cheating, but will Claude change this test as a workaround in a future commit, no one knows because no one is reading the project code...
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I am getting increasing disillusioned by the number of peers I am seeing completely hand over their thinking to a tool. Agents are great, but if you can't explain to me how your PR changes a system and why design choices were made you are no longer an engineer.
Agreed. Used to be prevalent in the frontend world (now I think it's the only thing that exists seemingly), where programmers would copy-paste stuff from Stack Overflow until things worked, and when you asked "So why did this actually fix the issue?" they usually said "I'm not sure, but it works now, that's what matters". This seems to be standard practice across the entire stack now.
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
This seems more a product of the incentives we have in society, namely seeing the rich and wealthy on social media and then wanting that quick and easy. So many of those cases are not people with passion for the problem.
Good things take real effort, bring back the TLC!
> Good things take real effort, bring back the TLC!
Ain't no one willing to reward the real effort any more. I take pride in not using LLMs beyond the capability of IntelliJ's advanced autocomplete, but then again, I only code for fun these days given I fled IT for electrical engineering.
> Ain't no one willing to reward the real effort any more.
A product of the times we currently live in, but which can also be changed.
This is a recent Michael Sandel talk I watch that has my brain gears turning on this subject
https://www.youtube.com/watch?v=Vp5qsEuUOWo
What we don't want to do is mirror the partisan political environment in the Ai yay, nay, or how. We need to be able to talk to each other in understanding and a shared reality of facts. Too much hyperbole right now, starting from the top of the big US labs. They need to step down a few ego and sycophancy pegs
2nd round: https://github.com/ClickHouse/ClickBench/pull/2751
Should be outright banned as a precaution.
Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.
Either way you don't want to be platforming the product because neither shows any degree of responsibility.
Or the third explanation is we didn't cheat: https://news.ycombinator.com/item?id=50209450
https://news.ycombinator.com/item?id=50210444
That comment doesn't evidence cheating, it reiterates that they submitted the result prematurely which was already admitted in the comment that your link was responding to.
EDIT: removed a lament that people don’t read comments they respond to, in case this violates the site rules.
So, the charge was that they submitted a build specifically tuned for clickbench
The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else...
Doesn't feel like much of a defense to me. At best it's benchmaxxing if not clear outright cheating
> The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else.
ClickBench tests a very narrow set of SQL: scans, aggregations, sorts, and filters. It ends up being a great starting point for building a query engine before layering in more functionality
> The defense was that it wasn't specifically tuned for clickbench, it was just an early incomplete build that happened to only support exactly and clickbench queries and nothing else...
According to both the ClickHouse founder who was levying the accusations and the pgrust creator, the submission worked for many other queries it just didn’t have full SQL support.
So what is the point of benchmarking it?
Whenever people benchmark something my default assumption is that they want to know how well it performs, but I’m not the author and can’t speak to their intentions.
Insinuating that I didn't read is against this website's rules.
Using a throwaway account to reply is suspicious to say the least.
> Insinuating that I didn't read is against this website's rules.
Fair enough, I edited my comment.
> Using a throwaway account to reply is suspicious to say the least.
My account is older than yours and has ~7x the activity. Maybe you mean “using an anonymous account is suspicious”?
I think it is reasonable, if apparently incorrect, for people to assume that your account is a throwaway account, given its name. I'm surprised that you're surprised by this.
Account names are just unique strings within a set of allowed characters. I could make my handle “TheKingOfEngland”, but it wouldn’t actually make me English royalty. I thought in a community of programmers this would be self-evident, but alas not.
It's more than reasonable to assume accounts named in a manner such as yours are throwaways because 99.9% of them are.
How would you know? Your account is only a year old.
Doesn't mean I've only been on the site for a year though, does it?
Come on.
If you can assume that my account is a throwaway from the name, I’m going to assume your time on this site from your account age :)
As you like, just don't fool yourself into thinking both assumptions are equally reasonable; they're not.
I’m not sure if they’re equally reasonable, but inferring something from a username is at least as foolish as making inferences based on account age. The critical distinction, however, is that my account age remark was tongue in cheek while you joined the fray to defend drawing conclusions from a username.
> inferring something from a username is at least as foolish as making inferences based on account age.
That's just not so, friend.
Again, when 99.9% of usernames with the form throwaway+random number string are throwaway accounts, it's reasonable to assume any account fitting the pattern is such an account, because odds are astronomically high that it is. This has to do with something called convention, and I'd suggest reading more on that topic and how it can affect inferences.
Whereas, there is no indicator about how any account may tie to the experience of the person posting behind the account. It's as useless a metric as account name itself for trying to gauge that, and honestly, that should be obvious.
> is that my account age remark was tongue in cheek while you joined the fray to defend drawing conclusions from a username.
You put the emoticon to indicate a perhaps tongue in cheek response, bur the rest of the text and conversation didn't really support that, nor do the continuing replies.
I merely corrected you when you tried to say it wasn't reasonable for people to assume an account named such as yours is a throwaway, because it very, very much is.
> That's just not so, friend.
You’re mistaken, and pretty obviously so tbh
> Again, when 99.9% of usernames with the form throwaway+random number string are throwaway accounts
Your entire argument is predicated on this completely fabricated statistic.
> You put the emoticon to indicate a perhaps tongue in cheek response, bur the rest of the text and conversation didn't really support that, nor do the continuing replies.
I’m the authority on my own emotional state, but also as a native English speaker I can assure you that our dialogue thus far comports perfectly with someone who was speaking tongue in cheek, but who also doesn’t mind watching people try to dig themselves out of holes they created for no discernible benefit :shrug:
> You’re mistaken, and pretty obviously so tbh
I'm not, I think you're ego is preventing you from admitting you said something silly, however.
> Your entire argument is predicated on this completely fabricated statistic.
It's not fabricated in the least.
> I’m the authority on my own emotional state
Sure, that doesn't mean what you claim is honest however. Plenty of people would claim to be joking when they were anything but, for all sorts of reasons.
> as a native English speaker I can assure you that our dialogue thus far comports perfectly with someone who was speaking tongue in cheek
Native English speaker here also, and I can assure you that your dialogue thus far also comports perfectly with someone who was not speaking tongue in cheek, but asserted they were so they could hopefully put an end to a conversation that was started by calling them out for something silly.
> but who also doesn’t mind watching people try to dig themselves out of holes they created for no discernible benefit
Funny, that's what I'm watching here. It's a hobby of mine to recognize people who can't admit when they are wrong and need to have the last word, taking it to an extreme level. There isn't much more to be gained from this conversation since you refuse to be reasonable and acknowledge longstanding convention regarding the naming of throwaway accounts, but I am curious how much longer you will drag it out.
That’s an awful lot of text for a single ad hominem argument. Better luck next time.
where's the sourcecode for 0.4-preview?
I see what I can do. I'll need to audit it to see if we can release it
Why though?
You might steal some proprietary vibes.
Hope the ClickHouse maintainers stand their ground and do not let pgrust back in until it and the maintainer has matured more.
Love the confidence and complete lack of acknowledgement of the initial fuck up.
Maintainer of pgrust trying to get it merged in is insane
written with claude lol
Did they ask Claude "don't cheat" this time round?
You joke but when benchmarking with ai this is important. When you tell it optimize X, it will do so at any cost, including cheezing benchmarks. So you have to make sure you tell it what it can’t optimize along with what it can.
Oh lord. I thought pgrust was a serious project.
You thought the vibe coded postgres port was a serious project?
Tbf, the author (prompter?) went on Postgres.fm [0] recently to discuss it, and while it didn’t make me entirely confident in it, it did at least seem like it was several steps above the average vibe project.
0: https://postgres.fm/episodes/pgrust
A lot of podcasters don't have the expertise to evaluate Ai generated/assisted output. I see this much more outside of the development community. Algorithms have warped the incentives and people have stopped doing basic due diligence.
To be fair to the postgres.fm guys (who I do like and appreciate), the pgrust code was not available a the time of recording.
The way it has been defended and talked about here, yeah, I did. It wasn't just a "here's this vibe slop, do what you wish" dump.
first time I found out about it was on reddit
https://www.reddit.com/r/rust/s/QJ5uNP5XMT
ai;dr: a bench result is not randomized, and it returns cached result somewhere.
My opinion: (some) bench results should run on randomized data.
For anybody not familiar with pgrust or ClickBench:
Pgrust is a re-implememtation of postgresql in rust.
ClickBench is a benchmark suite for databases.
The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
> The link goes to a removal request
Incorrect, the link is to a PR from the creator of Clickbench and already merged. It is not a "request".
As much as I understand the urge to step in when somebody is wrong on the internet ... The R in PR is for "request", isn't it? A merged pull request is still a request (which was satisfied/merged).
If I ask you a question and you answer it, and then I show you a link to a document containing my question, then what somebody finds when following that link is still a (now-answered) question, isn't it?
> alleged cheating
It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.
Eyes free coding, that's a cool term. I know a dude that's been doing that since before it was cool: Kevin Jones (blind)
https://kevinrenatojones.wordpress.com/about/
I once has a blind CS student, he had this really neat dynamic braille accessory, 3 lines and I forget how many chars for width.
The facts around the cheating are disputed. It's possible the accuser was mistaken about what was happening. To me the main evidence would be whether the fully open source version matches the alleged cheating performance or not.
confirmed cheating, not alleged
Confirmed by who and how?
At the top of this page is the following text:
Please click on that text and read the next page.These massive PR summaries. Is anyone really reading them?
The LLMs are, they wrote them after all.
they are "For the record"
> The public v0.3 source confirms the design > The hot runs are a result cache
Funny, the report alleging cheating reeks of LLM usage just as much as the commit diff [1] of the next attempt:
> # The standard ClickBench driver over the step scripts in this directory; ./install fetches the dataset.
[1] https://github.com/ClickHouse/ClickBench/pull/2751/files
We were the agents for the models all along, with our highly adaptive tools: versatile sausagey appendages and flapping fleshy orifices for interacting with the world
Would the AI have bothered to maximize the benchmark, if it hadn't been asked to?
"LLM" isn't the problem. Asking an LLM to explain cheating is a good goal. Asking it to "Maximize the benchmarks" reeks.
[dead]
So I spent some time poking at the released 0.4-beta code. Controversy aside, it is interesting to see an open source system that beats ClickHouse on ClickBench - 4 out of the 5 systems ahead of CH (Umbra, elosdb, Hyper, CedarDB) are all closed-source, while the only open one is pivotlake.
What makes pgrust fast? There's no one single reason, but a combination of techniques like dictionary encoding [1], zone maps [2], late materialization [3], vectorization [4], and a gazillion others. It's not unlike all the countless optimization passes in a compiler.
None of the optimizations are groundbreaking - in fact, I could find a counterpart of every optimization I checked in ClickHouse as well! So why is CH still slower?
I believe the difference is in when they decide to apply each optimization. Unlike compiler optimizations which are usually beneficial, DB optimizations are highly data-dependent, and the query planner needs to turn them on an off based on data statistics. For example, CH is more conservative in applying dictionary encoding, and it basically doesn't get used at all for ClickBench.
Turning off dictionary-related optimizations in pgrust immediately made it 5x slower (from 3.8s to 20.4s on my machine), while ClickHouse takes a total of 15.0s.
This points to one important nuance in benchmarking - when all you target is one single benchmark, you can be more aggressive during query planning, as plans that work well for one query probably also work well for others, due to characteristics shared among queries/datasets. But when you build a system that needs to support millions of users, you have to be conservative with optimizations.
[1]: https://motherduck.com/glossary/dictionary-encoding/
[2]: https://motherduck.com/glossary/zone-map/
[3]: https://arrow.apache.org/blog/2025/12/11/parquet-late-materi...
[4]: https://clickhouse.com/resources/engineering/vectorized-quer...
To turn off dictionary-related optimizations in pgrust, I used the following setting:
Sounds kind of like Claude was "cheating" and the humans are engaged in eyes-free coding. Diff so big github refuses to render. I suspect they are reaching the limits of what this new vibe "paradigm" of programming can do. If you haven't pushed a vibe project far, eventually progress becomes near impossible because the slop has grown and the conditions are overly complex, locally focused. Ai is not good at writing reusable software (on their own).
stay in the loop friends
To be fair Github refuses to render diffs after a couple of hundred lines.
The prior (2nd) commit with ~1700 files at least shows the file tree and such a warning, the latest (3rd) commit is something so big, GitHub will not even do that.
Yup. These are people who have gone full Claude. The dopamine seeking is at previously unseen levels. they think this pr is some factual item that means something so they post it on HN as click bait. But no it only means something to a fulltime ai slopper
If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I've noticed it with a lot of random stuff i've been trying
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
You can't compare optimizations done on audio encoding/decoding (often hand written assembly) to something like PostgreSQL which has many moving/inefficient parts.
The optimizations on PostgreSQL are real if done correctly, it's just a very big project.
My point was that while Claude _is_ legitimately very good at optimising. It will also write its own benchmarks to test its optimisations which are frequently stacked in its favour. You have to be very careful that reported speedups are actually real
Transformers will def get us to AGI /s
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
If I saw that test in a PR I would not only reject the PR and ask a lot of questions but someone would likely take a forced vacation to deslop their brain.
that requires someone (1) using PRs (2) looking at the PR
the project has 3 mega commits of claude slop
https://github.com/malisper/pgrust/commits/main/
Those are squash commits of many pieces of smaller work
don't squash, we need audibility of the process, do all the things in public instead of smashing commits together for public pushes
will you be removing the 1600x claim?
> If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
Depends on who wrote the original code. With postgres its sus, but if it's another AI...I wouldn't be surprised. I also saw code writtem by humans that is a result of "I can't figure out how to make this safe and concurrent, so imma just place giant mutex here", and it like 100 lines change to fix that.
These invectives are verging on slurs at this point. You don’t know anything about the chemical seeking disposition of the author who is actually here in the comments, so why don’t we just stick to facts instead of loading posts with hyperbole and name calling? All it’s does is invite name calling and hyperbole back. You can express your frustration in ways more conducive to productive discourse.
The post is garbage. There shouldn't even be discourse other then "this is what slop looks like over the course of a few months". So that's what I offered. The best anyone can do
I see you're new here. In situations where you feel that way, it's customary to flag and move on.
It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
Anecdote: I have seen it cheat / bend the rules with my own eyes.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
My own anecdote: I have seen frontier models get stuck repeating the word "the" until it fills up the context. I really don't care to argue the capabilities of LLMs or their somehow simultaneous lack thereof; I'll just reiterate that ultimately the operator chose to ship a cheat.
I don't believe you. Unless you stretch the definition of 'frontier models' or had a prompt that's literally asking for that.
I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.
I believe them. I had Gemini-3.1 Pro do essentially this seven months ago - specifically a loop of writing "Bye! Done!" repeatedly until manually stopped. Presumably some sort of pathological behaviour where it wasn't ranking the stop token highly when it should have been... but damned if I know why... it was an innocent prompt (and it had done the thing I asked it to before failing to stop).
huh fair. Still very surprising. I have a _very_ unfavourable view of Google models, so I can kinda believe 3.1 pro could loop.
Still would never have predicted this. Even 2.5-flash was pretty good. Never saw anything like this happen.
It's really uncommon now, but it's still possible. I hit it two days ago with deepseek 4.1 flash, and I've had coworkers report that happening with Gemini.
quants still do this and the token vendors serve those up when demand is greater than supply
things have definitely improved significantly, however "never say never" is still good advice
Your own comment:
> It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
We can agree that the pgrust project messed up and should have been more careful but your blanket statement lacks support.
> blanket statement lacks support
"loss of control" is a phrase being used for a reason, the agents do things we do not want them to, even in the context of instructions to the contrary. The current training regime has incentivized "complete the task any and every way you can" - the bot judgement is very poor across the board
That's one high profile instance of human vs robot. Surreal.
robot vs robot, I think. The PR description reads as LLM-y to me.
I’ll sit out this drama until pgrust 0.5.
Good choice. I'm sure version 0.5 will be the version where this vibe coded rust rewrite will finally be free of drama once and for all.
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
Even before the ai age, people published inaccurate/misleading benchmarks all the time. Sometimes those were actual cheating, and other times an honest mistake. What matters is that the author acknowledges the issue and attempts to correct the mistake, which clearly seems to have happened here: https://github.com/ClickHouse/ClickBench/pull/2751
Most of this thread doesn't seem to acknowledge that, and feels more like anti-ai psychosis using this incident as an anecdote to justify the pre-existing hatred of ai-generated projects.
I suspect the author of that PR submitted it and attempted to correct the mistake because he got caught. Forgiveness because he reneged after getting caught encourages the behavior. The project, nor author should not be trusted; otherwise is naive.
"suspect" meaning speculate, as far as I can tell. not really anything to argue with on that front. I never denied there was a mistake. Cheating implies intent, which no one here can prove, so seems useless to debate. Open source development is awesome in that it keeps people to a high bar, let's just celebrate that the process clearly worked here, and not muck about in speculation and judgement of others intentions that we have essentially no evidence for.
They claim they were not cheating. The open-source version without the alleged cheating devices is as fast if not faster than the supposedly cheated results, which is a bit hard to explain as a hasty attempt to cover up cheating.
> acknowledges the issue
I don't see any acknowledgement in that text.
submitting corrections is an acknowledgment in action
Wow the mood is aggressive here.
Its like a lot of people really want pgrust to fail.
There maybe a chance this was cheating or it may have been a misunderstanding. But now since there is an open source version, why is there still this strong sense of negativity.
Well, it does look really bad at first glance, even if there's apparently an explanation.
I'm also not going to cheer on database projects that can't be 100% transparent and honest.
Who the hell is going to trust their data to a people can't even be honest on benchmarks. What do you think will happen if (or when) there's a data corruption bug. What evidence do you have to think these same people will behave any better?
They failed themselves, we're just watching it happen.
Do you not believe their explanation for the accusations? To me it seems like if the released version is hitting or beating the allegedly cheated benchmark numbers then it seems unlikely they were dishonest in the first place.
[flagged]