> One model drafts a result, an independent read-only critic from a different model family reviews it
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
My editor and gatekeeper use like the same model Opus 5. Different prompts and a kind of different input data. The gatekeeper receives the fact check results next to finished text. In the same time the editor already delivered them.
As far as I remember over the entire period he removed 27 posts out of 187 that went through him.
So I believe that different manufacturers are not mandatory. What matters I guess is not the difference in models but... the fact that the critic has a different input and doesn’t have their own text that needs to be defended. That's the point
> the fact that the critic has a different input and doesn’t have their own text that needs to be defended.
Different inputs are one point, but there is another problem: lack of diversity
Models from the same maker, share the same training and the same implicit bias. It is like if both reviewers had the same gender, race, and studied at the same university, and just got different book day before. Add fresh immigrant from rural asia, you get VERY different opinions, even with the same input book...
Plus practical aspects, Opus 5 is sometimes way too creative which is good for writting. GPT Sol is complete oposite, it is obsessed with crossing every T and verifying every dot. It complements Opus as reviewer!
If opus gets security sensitive questions, gets downgraded to sonnet and againdown to haiku, the same model will hit the same security block, and will not catch the issue. Model from another lab will very likely catch this.
Plus anthropic models love to smell their own farts, load bearing seems are fantastic...
I would answer why was a single model enough for this? The fact check provides a kind of explicit written list of what cannot be asserted. At the same time gatekeeper compares the text with this list. I would say this is not about taste or judgment it’s about comparing two documents. In addition some of the removed items are direct violations of this list like the post called the model "a top‑tier model" even though the fact check by the way explicitly prohibited presenting circulating benchmarks as established facts.
As regarding the degradation to Sonnet and Haiku I don't have yet any data.
If you’re a copilot cli user (we exist) you know that the auto mode leaves something to be desired. It only changes models at the start of a session or after compaction. Most dev tasks are now sub agent heavy and there is no model routing on the sub agents.
I do notice that there is no mention of effort levels in the comparisons. 5.6 Luna Max is really good and really cheap. A 5.6 Sol high orchestrator with 5.6 Luna max is cheap, has frontier level performance, and is faster than Sol alone. This can be accomplished with simple agent instructions. Looking forward to running my own benchmarks on HydraFusion to see how it fares. Gone are the days of a single model doing all of the work it seems, unless the work requires no tool calls.
I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.
This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.
Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics:
- tweaks few decode settings like slightly higher temperature
- detects when model gets stuck and tells it to "go on"
- detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"
And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.
Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.
I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things:
- either opus doesn't need it
- or it's already done behind the scenes.
In my view, the useful like for like comparison is to the entire system that people actually use. Nobody uses a "naked model", so what is the point of these benchmarks that use them in that way?
I think the benchmarks should be trying to use realistic harnesses for both proprietary and open weight models.
Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.
I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the model, then there should be some other detectable improvement, like to speed or cost.
I don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses.
It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.
I agree that all else equal, the more comparisons the merrier. But IMO, there has been way too much focus on benchmarks for the bare models, when it seems to me that what actually matters is what you can do with the technology, and that is never limited to what a bare model can do.
I haven't written one. You can easily replicate it if you wish just based on my comment and a day spent with Claude Code.
In fact that is how I got the idea. There is a 4 month old post on SWEbench github that claimed 20 point boost (beating opus as well) for another small Qwen model when it's traffic was sent via a similar proxy and claude code was used as the harness.
The author didn't post their full code, but he did post all sorts of logs showing which features of his proxy were activated during which run.
I wanted to exclude the harness and validate his claims regarding only the proxy's impact so I reimplemplemented a subset of his features that seemed to give me the biggest "wins" with the smallest amount of work. These are the ones I list in the parent.
It took one afternoon last week plus the runs that took few days.
I saw about 10% better results.
As I mentioned, it is a subset of tasks, not the entire set, but results so far are very encouraging.
Since then I implemented many more features in the proxy (best of N for example), but i haven't had time to measure the impact yet. His proxy also had best of N, but that feature wasn't activated in on the runs he reports.
i've been using adversarial critique and reviews for many planning, solution design and implementation steps inside workflows.
It's so effective and helps catching so many design flaws, implementations misses etc ... that i'm wondering how people manage to build complex/large projects with agents without this kind of process. Well, i actually built this thing because i couldn't get good results so i had to find a way.
I'm gonna open source the whole thing but it needs some cleanup, there's a basic landing page here https://kodfactory.com if anyone wants to be notified when it's released on github. Yeah i know, the world really needs another software factory :-)
Why? I feel it kinda implies what the approach is. Could definitely be worse, strongly prefer this over just giving the thing a random-ass name like Laguna.
> Hydra (sometimes stylized as HYDRA) is a fictional terrorist organization appearing in American comic books published by Marvel Comics.[…] Hydra is taken over and turned into a neo-fascist international crime syndicate by Baron Wolfgang von Strucker.
With all due respect, but I'm not letting some newfangled comic taint/monopolize a perfectly usable name that's been used to describe many-headed-thing (water snake) for over two millennia.
This looks very similar to what I've seen with a different harness - https://github.com/AMAP-ML/LongHorizon-Harness. It's critique pattern looks like another version of lh-harness without looping toward an overarching goal
I have a custom agent that I call my "G.S.D." (Get Stuff Done) agent.
It is explicitly not a foreman, task routing, or an orchestrator agent. It has a bias toward direct action and is instructed to only delegate when necessary.
I've found that this approach yields significantly faster results, without much of a quality trade-off, than an agent whose primary impulse is to delegate.
In contrast, HydraFusion starts with a task routing step, then sequential planning, execution, and review stages. My guess is that this workflow is best for people who are prioritizing cost over speed for the same level of quality.
> In controlled offline evaluations, HydraFusion’s selective coding workflows matched or exceeded the evaluated Opus 5 baseline
Opus 5 (in practice) is not a good baseline to compare against.
I'm shocked that they chose to primarily compare against Opus 5 in all the article's charts. It's pretty disingenuous that they're claiming "frontier" quality, but didn't compare against Fable or Sol.
I would imagine they did not test against Fable because Microsoft and GitHub (like many big companies) have internally given the instruction not to use this model, because of the data retention policy.
They did also compare to Sol and the comparisons are still favorable. However, they were most favorable comparing to Opus 5 because of the cost judging by the charts.
> One model drafts a result, an independent read-only critic from a different model family reviews it
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
[1] - https://github.com/t3rmin4t0r/critique-evals
My editor and gatekeeper use like the same model Opus 5. Different prompts and a kind of different input data. The gatekeeper receives the fact check results next to finished text. In the same time the editor already delivered them. As far as I remember over the entire period he removed 27 posts out of 187 that went through him. So I believe that different manufacturers are not mandatory. What matters I guess is not the difference in models but... the fact that the critic has a different input and doesn’t have their own text that needs to be defended. That's the point
> the fact that the critic has a different input and doesn’t have their own text that needs to be defended.
Different inputs are one point, but there is another problem: lack of diversity
Models from the same maker, share the same training and the same implicit bias. It is like if both reviewers had the same gender, race, and studied at the same university, and just got different book day before. Add fresh immigrant from rural asia, you get VERY different opinions, even with the same input book...
Plus practical aspects, Opus 5 is sometimes way too creative which is good for writting. GPT Sol is complete oposite, it is obsessed with crossing every T and verifying every dot. It complements Opus as reviewer!
If opus gets security sensitive questions, gets downgraded to sonnet and againdown to haiku, the same model will hit the same security block, and will not catch the issue. Model from another lab will very likely catch this.
Plus anthropic models love to smell their own farts, load bearing seems are fantastic...
I would answer why was a single model enough for this? The fact check provides a kind of explicit written list of what cannot be asserted. At the same time gatekeeper compares the text with this list. I would say this is not about taste or judgment it’s about comparing two documents. In addition some of the removed items are direct violations of this list like the post called the model "a top‑tier model" even though the fact check by the way explicitly prohibited presenting circulating benchmarks as established facts. As regarding the degradation to Sonnet and Haiku I don't have yet any data.
https://arxiv.org/abs/2601.14351
If you’re a copilot cli user (we exist) you know that the auto mode leaves something to be desired. It only changes models at the start of a session or after compaction. Most dev tasks are now sub agent heavy and there is no model routing on the sub agents.
I do notice that there is no mention of effort levels in the comparisons. 5.6 Luna Max is really good and really cheap. A 5.6 Sol high orchestrator with 5.6 Luna max is cheap, has frontier level performance, and is faster than Sol alone. This can be accomplished with simple agent instructions. Looking forward to running my own benchmarks on HydraFusion to see how it fares. Gone are the days of a single model doing all of the work it seems, unless the work requires no tool calls.
I'm questioning their results. It doesn't take much to beat the frontier in single benchmarks if one puts extra software between the model and the harness.
This is also a reason why comparing "naked models" for which weights are available and frontier where providers can do whatever they want behind the scenes is unfair.
Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics: - tweaks few decode settings like slightly higher temperature - detects when model gets stuck and tells it to "go on" - detects responses cut in the middle, empty responses that contain only reasoning, formats not passing verification etc and tells the model to "try better"
And that is it. 10% more. I admit on a subset of tasks, but results are results, even on a subset.
Yeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
It is useful to compare like for like.
Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.
I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doesn't need it - or it's already done behind the scenes.
In my view, the useful like for like comparison is to the entire system that people actually use. Nobody uses a "naked model", so what is the point of these benchmarks that use them in that way?
I think the benchmarks should be trying to use realistic harnesses for both proprietary and open weight models.
Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.
I don't quite understand your point. Why are you moving those capabilities from one model to another, or improving the built-in capabilities of a model, what is the goal? If having the capabilities in the model itself improves the overall capability, then using that better model with the same harness should achieve better results. Or if the capabilities are the same, but they've been moved from the harness into the model, then there should be some other detectable improvement, like to speed or cost.
I don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses.
It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.
I agree that all else equal, the more comparisons the merrier. But IMO, there has been way too much focus on benchmarks for the bare models, when it seems to me that what actually matters is what you can do with the technology, and that is never limited to what a bare model can do.
> Specifically, all it took to boost Qwen3.8-27B to get 10% more points on SWEbench Pro and Terminal Bench 2.0 with a proxy that has just these basics
Do you have a link/paper for this you could share?
I haven't written one. You can easily replicate it if you wish just based on my comment and a day spent with Claude Code.
In fact that is how I got the idea. There is a 4 month old post on SWEbench github that claimed 20 point boost (beating opus as well) for another small Qwen model when it's traffic was sent via a similar proxy and claude code was used as the harness.
The author didn't post their full code, but he did post all sorts of logs showing which features of his proxy were activated during which run.
I wanted to exclude the harness and validate his claims regarding only the proxy's impact so I reimplemplemented a subset of his features that seemed to give me the biggest "wins" with the smallest amount of work. These are the ones I list in the parent.
It took one afternoon last week plus the runs that took few days.
I saw about 10% better results.
As I mentioned, it is a subset of tasks, not the entire set, but results so far are very encouraging.
Since then I implemented many more features in the proxy (best of N for example), but i haven't had time to measure the impact yet. His proxy also had best of N, but that feature wasn't activated in on the runs he reports.
i've been using adversarial critique and reviews for many planning, solution design and implementation steps inside workflows.
It's so effective and helps catching so many design flaws, implementations misses etc ... that i'm wondering how people manage to build complex/large projects with agents without this kind of process. Well, i actually built this thing because i couldn't get good results so i had to find a way.
I'm gonna open source the whole thing but it needs some cleanup, there's a basic landing page here https://kodfactory.com if anyone wants to be notified when it's released on github. Yeah i know, the world really needs another software factory :-)
They should probably focus on “project hydrastable infrastructure” instead
This has got to be the worst project name I've seen all year.
I think the name probably has to do with this research from Microsoft (same people involved and a lot of similarities in the way it's conceived.
HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools : https://arxiv.org/pdf/2605.17106
Seems more like many heads -> one head and someone thinking it's clever.
But could be your thing too obviously.
Considering the people thanked in the article are the same who wrote the paper I linked, and the similar mechanisms, it's more than likely
Why? I feel it kinda implies what the approach is. Could definitely be worse, strongly prefer this over just giving the thing a random-ass name like Laguna.
https://en.wikipedia.org/wiki/Hydra_(comics)
> Hydra (sometimes stylized as HYDRA) is a fictional terrorist organization appearing in American comic books published by Marvel Comics.[…] Hydra is taken over and turned into a neo-fascist international crime syndicate by Baron Wolfgang von Strucker.
With all due respect, but I'm not letting some newfangled comic taint/monopolize a perfectly usable name that's been used to describe many-headed-thing (water snake) for over two millennia.
I think the problem here is that we're running out of names. Which is ironic after calling everything copilot or office 365
This looks very similar to what I've seen with a different harness - https://github.com/AMAP-ML/LongHorizon-Harness. It's critique pattern looks like another version of lh-harness without looping toward an overarching goal
Auto model selection has been basically unusable for me, so I'm really unsure how much to trust the post's narration on this...
Oh well, can't wait to be harassed by management to try and use it anyways...
I have a custom agent that I call my "G.S.D." (Get Stuff Done) agent.
It is explicitly not a foreman, task routing, or an orchestrator agent. It has a bias toward direct action and is instructed to only delegate when necessary.
I've found that this approach yields significantly faster results, without much of a quality trade-off, than an agent whose primary impulse is to delegate.
In contrast, HydraFusion starts with a task routing step, then sequential planning, execution, and review stages. My guess is that this workflow is best for people who are prioritizing cost over speed for the same level of quality.
From the article:
> In controlled offline evaluations, HydraFusion’s selective coding workflows matched or exceeded the evaluated Opus 5 baseline
Opus 5 (in practice) is not a good baseline to compare against.
I'm shocked that they chose to primarily compare against Opus 5 in all the article's charts. It's pretty disingenuous that they're claiming "frontier" quality, but didn't compare against Fable or Sol.
I would imagine they did not test against Fable because Microsoft and GitHub (like many big companies) have internally given the instruction not to use this model, because of the data retention policy.
They did also compare to Sol and the comparisons are still favorable. However, they were most favorable comparing to Opus 5 because of the cost judging by the charts.
Well, saying it was a shitshow compared to … is just not good marketing i guess
that's crazy
Maybe I'm just 2026's Dropbox guy, but... yes, and?
[flagged]
[dead]