> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.
> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
All benchmarks are vulnerable to manipulation.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.