These general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks.
For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?
The externalities have to be dealt with using human judgment. Ai can help organize data to make better judgements but that use case is taken by chat based AI.
By the time, we want to code, the scope has been limited pretty greatly to the point that my rule to the Ai is “don’t make any decisions outside of the plan, if something in the plan turns out to be untrue or cannot be implemented after 2 differing attempts stop and ask for clarification”.
Sub frontier level Ai can circle endlessly thanks to a bad assumption, but a human can cut through that by coming at the problem sideways.
Its a good point that there is benefit to using general intelligence models to figure out what code to write and to write specs. Implementation plans as well as long as the general intelligence is also great at coding. But you would actually prefer here the model that performs best in your business domain rather than best at general intelligence.
The Opus frontier model will spin up a cheaper Sonnet subagent to explore the code base. Similarly we can have a Pareto optimal subagent write most code and verify it.
Are you approaching it like programming? I first design the program I want it to write, and then let the LLM implement it. All the important decisions it has to make are contained in the design. The LLM only has to know programming and a little bit of algorithms to implement it.
How often you needed the coding AI know all the trivia about some metal band? Like who are the members, songs , years etc. You can't even be sure if the facts are correct . This are good for a generalist model where you say remember some scene from music video and the AI will find the video for you.
Sure some world knowledge like things are affected by gravity, how people function, how a society function I think is needed for a coding AI but not trivia.
How often do the cousins of Louis the 5th of France impact on coding decisions? I can't see the argument for a completely generalist model being best for _coding_ in all situations.
They probably don't. But some words acquire their full sense meaning from historical and-non-programming context.
Words that are or were fairly common in tech circles — "bailiwick", "gatekeeper", "daemon", "boundary", "infer", "method".
It's true to say that most of these words have some specific meanings in tech, but you don't necessarily want the LLM to have only those narrow meanings when you are talking to it. A broader grasp of the word means the LLM won't go down a specific pathway before understanding you by analogy. That broader grasp comes from non-programming understanding.
Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).
Are these comparisons based on API price? If so, then the OpenAI/Anthropic subscriptions (which are a 15-40X discount on model pricing) turn the comparison into a bloodbath, even more in favor of the closed source models.
In theory, yes. In practice, this comes with a few caveats.
First, OpenCode and Command Code offer you basically 6x-10x usage for some models (not all, though, look at the exact rates per model). That's a very similar model.
Second, while Anthropic subscriptions on paper offer the most generous usage per dollar spent [1], this comes with a few caveats. First, they are the most expensive of all models in cost/task. Second, if you use Claude Code non-interactively (claude -p "..." or SDK/ACP mode), they apparently charge you 70% more usage; you have to use their apps directly and interactively to get the full benefit of a subscription. Also, Anthropic has the lowest cache TTL at 5 minutes, which can really hurt for interactive use. Contrast that with OpenAI's default of 30 minutes (and according to a recent post by Tibo, apparently 12 hours for subscriptions, though I haven't verified it myself) or the fact that DeepSeek caches prompts for three days (which is a major reason for their extremely high cache hit rate and how cheap they are in practice), and it doesn't look like THAT great of a deal in comparison anymore. It is of course still good value for money, just not as great as it may look at first glance.
[1] At least before the recent usage bump to OpenAI's Pro plans, though that doesn't seem to include the Plus plan if that is what you are on.
Hey, thanks for the shout-out! (I'm one of the cofounders of Synthetic.new.) I think I recognize you from Discord :)
Yep, on subscription plans DS4.1-Flash-1017 feels more or less infinite. I still reach for Astra at the high end of tasks but DS is great for cheap "repeatedly crawl the entire codebase and find query optimization" type tasks. I also use a lot of K3.1 Pro lately although admittedly it's a lot more expensive.
Also personally I find that the closed tiny models (Haiku, Luna) feel a lot less capable in practice than their benchmark scores indicate relative to DS.
That being said the closed labs own the high ground of intelligence, nothing really compares to Astra in my opinion in terms of sheer "it will achieve any task you give it," Kool-Aid-Man running through walls style capability. It's quite expensive but it's also very very capable.
I just checked out your Go plan and noticed the GLM 5.3 Flash credits are pretty low compared to the other flash models. Is there any plans to change that?
I think the biggest money unlock for most people is just to not use the largest models. Luna / Deepseek are a dollar or two a day for coding. Accidentally (or purposefully set a different model) and boom the cost shoots up. I think most people could slash their budgets by 90% just by choosing cheaper models most of the time.
> I think the biggest money unlock for most people is just to not use the largest models. Luna / Deepseek are a dollar or two a day for coding.
If someone is in a position to use the personal subscriptions, the $20/month plans are about a dollar per work day. I'd rather have the $20/month plan for Opus or Sol 6.1 than deal with all of the extra churn and babysitting from Luna or Deepseek.
Isn't the obvious answer that closed labs are massively subsidizing your usage using investor dollars? That's nice for today's users, but that isn't sustainable long term.
I'm just going to paste my comment from another post since it is applicable here too, maybe even more so.
I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.
The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.
It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.
DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.
For context I do equal parts sysadmin/devops and having them write software
I run with DSH, I've found it crazy customisable with the Cordis framework and I feel like I understand how the components in it work. So I'm actually able to articulate to the agent what I want for the customisations. I Tried pi and OMP but I couldn't get exactly what I wanted.
As for the session maintenance, I have it configured to trigger a compaction when it ends a 'debate' with >25% of context used but before the compaction it puts what it thinks are key points into an MD file. The approach has worked so far in terms of stopping the agent from getting silly with too much context and retaining information. These flash agents get too flakey for my liking past 25%
I wrote the prompt to get the agent to write the plugins for me lol
DSH has been fantastic, the customisation is great, like I mentioned before. What has got me to stick with it though is the way it presents information like diffs, subagents, tool calls, thinking and sessions. It's nothing revolutionary but everything is where I want it to be and provides just enough information
Not that it actually matters here, but my last generation flagship gpu is currently listed on newegg for exactly twice as much as the launch MSRP that I paid. Even decent-enough/established ebay listings for used ones are +$500 lol
Come to think of it, my Qwen box has gone up in value at a rate of $235 / month since I bought it. I guess I could sell it + it would retroactively pay for my Anthropic subscription.
IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve.
I have real world data here.
On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.
This is totally suitable (speed wise) for coding.
This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).
No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token.
Given that costs for "consumer-ish" GPUs are rapidly rising, I wonder if you'd also have to distinguish two "tiers" of local inference by now: Cost if you bought the hardware a year ago and therefore already own it vs cost if you have to buy the hardware new.
The article notes that this is mainly because the cheap models tend to be distillations of the frontier closed source models, and the developers of the original closed source frontier models have an edge in training their cheaper versions (since they have the source). I'd add that they also have funding, and want to blitzscale. OpenAI / Google can offer a cheap model to get more training data (plan dependent), or have a lot of capacity due to training requirements.
It is also evidence of commodicisation / brutal competition. I wonder if they cut costs because there is stiff competition from open weight models (and arguably we are in the blitzscaling phase where they try to corner the market, not the enshitification stage).
While the distillation argument is brought up again and again, I find it very dubious. This is not to say that it doesn't happen, but I have my doubts that it is a limitation. DeepSeek V4.1 Flash, for example, can reason itself through some pretty complex mathematical problems. Definitely not millenium-prize level problems and frontier models are superior for research-level mathematics, but it is still surprisingly close even for some of that [1, 2].
Where it differs from the larger models is that it has less knowledge; respectably as it performs for its size, 552B parameters are still a lot less than the multiple trillions that frontier models have. Sol/Astra/Opus/Fable may know the solution to the problem directly or to a very adjacent problem, while DeepSeek has to work its way through it from first principles and then sometimes fails (because LLM reasoning is really a heuristic-guided search). But this is also a failure mode that I have seen with Luna.
I don't think I am going to try to beat them at this game anymore. I've got a machine in my closet that is both collecting dust and accruing value. A very maddening state of affairs.
At this point, trying to beat a major provider is like running your whole house off grid when the alternative is 12c/kWh that is already wired to the premises. You might win if you make it your only job for years on end and have a very specific lifestyle that is tolerant to weird tail risks (assuming you aren't willing to go back to the grid in an emergency).
... and yet ... with codex on my subscription it gets nowhere. Clearing out the entire monthly quota in half a week happens every time now. Didn't use to.
So what's the point? After the codex <> web quota merge next month it'll be useless ...
This is so wrong. I can tell that the poster never used deepseek 4.1 flash for substantial tasks. To me it's absurd how much more people pay for Claude & Co. for even complex software engineering tasks (not talking about finding the cure to cancer here).
Just look at the caching prices and expiration times... Did you know that deepseek caches are alive for many hours usually between 1 and 3 days(!)?
For a fraction of the price of openai or anthropic?
I'm using Claude all day at work and deepseek privately for very similar tasks. The price/performance difference is extreme.
Even as the article points out, this fluctuates constantly. In a few months, we're guaranteed to get a new batch of open models, and the balance will shift again.
What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.
That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.
That's to be expected since all the proprietary models were just released. Give it six months and the Chinese labs will distill from the proprietary models like they did last time.
The article itself goes against this intuition, open models have taken up as much as 40-50% of frontier for capable/affordable. Right now is very much a low point. The fact there's a lot of churn as new models are released is not unexpected, I think.
If required, so far I'm not impressed with newest proprietary. i have a claude sub mostly rotting at this point, so i come back from time to time until the year long sub expires, but at this stage i will not renew and see no point in doing so.
Have you tried Opus 5.5 and Fable 5.1? They're still better than any of the available open-weights models. (And rumors are Fable 5.5 is coming out tomorrow or next week.)
I have a Claude $200 (Ultra these days) plan and use Opus 5.5 for work with ultracode workflows and all the rest every day. I also use DeepSeek V4.1 Flash 1107 through a separate subscription plan with Pi.
Using DeepSeek I've made the Wayland compositor I use every day. It's finished now in the sense that it doesn't crash or have any bugs, nor can I think of any features to add or anything to change.
When somebody else is paying, I'll absolutely just throw Opus 5.5 at it. Who cares. It gives the best chances that the output is what I want, at no extra cost to me. But for my personal projects, using DeepSeek in Pi with a workflow extension is competent.
Sorry for throwing lots of text at you. All of this to say there's more to it than just which model is more intelligent. There's also speed, cost, and "good enough". Perhaps the problems I'm working on right now are just not complicated enough for "Opus is better" to be a real selling point.
Have used 5.5, ran usual battery of tests, and the downhill death spiral in a science/chemistry context continues. Code wise, it is better than open weight in several ways, but you really have to work hard to fight it from creating such a heavily abstracted mess it becomes unmaintainable for humans. Requires too much handholding for me. It's scientific reasoning is more confident as well, which would be great if it were also more correct.
Open weight models are generally more humble when working at fronteir, seek human input more reliably on thi gs they have no buisness deciding on, and can be trusted to not fuck up codebase in some way. Also, the way claude talks by default and tends towards even with all the "use this standard or that" tricks, grinds my gears in a way that drive me postal.
So, all in all, the shit prose, the constant oversight robbing me of attention better spent elsewhere, and high cost, and seemingly correct and difficult to catch errors on first pass make avoidance/non-reliance a nobrainer.
When dumb models are right, they're right, because likely its a solved problem elsewhere, and when they're wrong its quite transparent. Claude and fable are extremely good at spinning resilliant bullshit that can take quite some mental labor to verify and disprove, usual problem is flawed or deeply i correct assumptions on top of which then relativly sound reasoning and evidence is layered, or otgerwise weird contradictory decisons made "because i thought it would be better this way" that waste 48 hours of gpu time on an invalid model because claude fucked the one small part entrusted to it, which was extremely well defined and reasoned/backed by literature and it just made an executive decision to deviate without consulting or warning about it.
IMO there's no point commenting on it unless you've actually tested using Opus 5.5. You can't assess the status quo based on even one generation prior.
It’s not, of course; you’re going to act on your own requirements as you should.
But I think it becomes a little more relevant when making a price-performance comparison such as the article makes between “open” models and “closed” models.
Because the reader is invited by the term “price-performance” to think about efficiency, when in reality the results are being skewed by the fact that the user is not the only one paying for the service: investors are chipping in fairly directly for every token, at inference time.
Users are going to continue to do their own thing for their own reasons, and they should; that’s the market at work. But this is not an efficient market where meaningful comparisons can be made; all of the major closed providers are loss-leading.
Opus 5.5 is currently cheaper than Deepseek V4.1 flash (with APIfied subs, but who cares, that's the price for individuals & startups, unsure why everybody seems obsessed about "API is the price" while it's genuinely not except for Enterprises specifically) - It's quite hard to justify switching outside of the privacy factor.
For us who spends more than $150K tokens a month with less than $5K spending in subs, it's really hard to justify the switch and it's even senseless (for now).
DeepSeek also has APIfied subs, and they have competition in-between so customers can have a great price.
I had a check on artificialanalysis.ai and it shows a 16% reduction in token and 200% increase in time for Opus 5.5 Max compared to DeepSeek V4.1 Flash Max. I don't think the data is very accurate for my real life workload, but for any workload it represents I would think Opus having little advantage.
True but let's think about whether this actually matters in practice going forward, given that today, Deepseek/any OS-27B variant can do all the things that most devs need it to.
Let's scale this out 3 years into the future:
- Opus 6.7 costs an arm and a leg to run for 2 prompts
- Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive
- As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)
Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?
... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.
Despite its simplicity, price-performance is not the only metric; this obsession is a typical consequence of extreme overextension of the Friedman doctrine, and it is why so much software and hardware has been qualitatively shitty even when it matters that it’s not. That something can be measured doesn’t automatically make it preferable as a decision-making basis over intangibles.
It’s pretty obvious that local AI has a higher TCO if you look only at money. But there are other things to be considered.
These general intelligence/capability benchmarks don't apply to your specific task. If you are coding, you want just benchmarks for coding. I don't see performance+price at this benchmark for coding. When you look at just coding, DeepSeek is Pareto optimal for a good portion of coding benchmarks. For example here, although even this is very flawed because it only benchmarks max/xhigh effort: https://artificialanalysis.ai/agents/coding-agents
Closed models might be better generalists because they don't have that much constraint on size and come from well funded labs that can afford to better train for generalization. If a model's actual goal is to be open then smaller is better.
I see this take frequently; that coding does not require general intelligence. Perhaps I am in the minority, but rarely do I find myself in coding situations that are completely divorced from other subject areas. The code is modeling or interacting with the world, either through business processes, or (in my case) simulating the physical world, and I need the agent to have some working knowledge of not just the code, but also physics, materials, or social conventions. Are you all really able to just "write code" without dealing with all the externalities?
The externalities have to be dealt with using human judgment. Ai can help organize data to make better judgements but that use case is taken by chat based AI.
By the time, we want to code, the scope has been limited pretty greatly to the point that my rule to the Ai is “don’t make any decisions outside of the plan, if something in the plan turns out to be untrue or cannot be implemented after 2 differing attempts stop and ask for clarification”.
Sub frontier level Ai can circle endlessly thanks to a bad assumption, but a human can cut through that by coming at the problem sideways.
Its a good point that there is benefit to using general intelligence models to figure out what code to write and to write specs. Implementation plans as well as long as the general intelligence is also great at coding. But you would actually prefer here the model that performs best in your business domain rather than best at general intelligence.
The Opus frontier model will spin up a cheaper Sonnet subagent to explore the code base. Similarly we can have a Pareto optimal subagent write most code and verify it.
Are you approaching it like programming? I first design the program I want it to write, and then let the LLM implement it. All the important decisions it has to make are contained in the design. The LLM only has to know programming and a little bit of algorithms to implement it.
How often you needed the coding AI know all the trivia about some metal band? Like who are the members, songs , years etc. You can't even be sure if the facts are correct . This are good for a generalist model where you say remember some scene from music video and the AI will find the video for you.
Sure some world knowledge like things are affected by gravity, how people function, how a society function I think is needed for a coding AI but not trivia.
How often do the cousins of Louis the 5th of France impact on coding decisions? I can't see the argument for a completely generalist model being best for _coding_ in all situations.
They probably don't. But some words acquire their full sense meaning from historical and-non-programming context.
Words that are or were fairly common in tech circles — "bailiwick", "gatekeeper", "daemon", "boundary", "infer", "method".
It's true to say that most of these words have some specific meanings in tech, but you don't necessarily want the LLM to have only those narrow meanings when you are talking to it. A broader grasp of the word means the LLM won't go down a specific pathway before understanding you by analogy. That broader grasp comes from non-programming understanding.
They do, to the extent that general intelligence training is what gives you the ability to talk in broad natural language to a coding agent.
Words only have their meanings through general context.
Yes, 100% agree. Without more details on how they got their data, it's very hard to understand why they don't value DS Flash 4.1 as much as pretty much every other analysis I've seen (and my own experience).
[dead]
Are these comparisons based on API price? If so, then the OpenAI/Anthropic subscriptions (which are a 15-40X discount on model pricing) turn the comparison into a bloodbath, even more in favor of the closed source models.
In theory, yes. In practice, this comes with a few caveats.
First, OpenCode and Command Code offer you basically 6x-10x usage for some models (not all, though, look at the exact rates per model). That's a very similar model.
Second, while Anthropic subscriptions on paper offer the most generous usage per dollar spent [1], this comes with a few caveats. First, they are the most expensive of all models in cost/task. Second, if you use Claude Code non-interactively (claude -p "..." or SDK/ACP mode), they apparently charge you 70% more usage; you have to use their apps directly and interactively to get the full benefit of a subscription. Also, Anthropic has the lowest cache TTL at 5 minutes, which can really hurt for interactive use. Contrast that with OpenAI's default of 30 minutes (and according to a recent post by Tibo, apparently 12 hours for subscriptions, though I haven't verified it myself) or the fact that DeepSeek caches prompts for three days (which is a major reason for their extremely high cache hit rate and how cheap they are in practice), and it doesn't look like THAT great of a deal in comparison anymore. It is of course still good value for money, just not as great as it may look at first glance.
[1] At least before the recent usage bump to OpenAI's Pro plans, though that doesn't seem to include the Plus plan if that is what you are on.
There are open model subscriptions with discounts as well (opencode go, commandcode, ollama cloud, synthetic, ..)
edit: and of course from first party labs as well, like xiaomi, z.ai, kimi etc
Hey, thanks for the shout-out! (I'm one of the cofounders of Synthetic.new.) I think I recognize you from Discord :)
Yep, on subscription plans DS4.1-Flash-1017 feels more or less infinite. I still reach for Astra at the high end of tasks but DS is great for cheap "repeatedly crawl the entire codebase and find query optimization" type tasks. I also use a lot of K3.1 Pro lately although admittedly it's a lot more expensive.
Also personally I find that the closed tiny models (Haiku, Luna) feel a lot less capable in practice than their benchmark scores indicate relative to DS.
That being said the closed labs own the high ground of intelligence, nothing really compares to Astra in my opinion in terms of sheer "it will achieve any task you give it," Kool-Aid-Man running through walls style capability. It's quite expensive but it's also very very capable.
I've been subscribed to Synthetic for months and it's a great service [I didn't put you first in my list because I don't want to shill :^)]
I just checked out your Go plan and noticed the GLM 5.3 Flash credits are pretty low compared to the other flash models. Is there any plans to change that?
I think the biggest money unlock for most people is just to not use the largest models. Luna / Deepseek are a dollar or two a day for coding. Accidentally (or purposefully set a different model) and boom the cost shoots up. I think most people could slash their budgets by 90% just by choosing cheaper models most of the time.
> I think the biggest money unlock for most people is just to not use the largest models. Luna / Deepseek are a dollar or two a day for coding.
If someone is in a position to use the personal subscriptions, the $20/month plans are about a dollar per work day. I'd rather have the $20/month plan for Opus or Sol 6.1 than deal with all of the extra churn and babysitting from Luna or Deepseek.
[dead]
Isn't the obvious answer that closed labs are massively subsidizing your usage using investor dollars? That's nice for today's users, but that isn't sustainable long term.
The fact that open models are even close despite an unfair playing field is telling
As stocks fall and the investment money dries up, it'll be easier and cheaper to buy hardware to run local models
I'm just going to paste my comment from another post since it is applicable here too, maybe even more so.
I've had really great results with cheap flash agents using a system that has a parent agent that spawns 2 sub agents. They remain active throughout the whole session and don't lose context.
The 2 sub agents get the exact same prompt and tools as each other and they independently answer but before responding back up to the parent they exchange their findings which must be backed up by a citation and then debate their findings and citations repeatedly until they can agree on correctness of all points. They are also heavily encouraged to never assume and always verify with the prompt and tools and guards where they can be realistically enforced.
It churns through tokens but tokens are crazy cheap with some of the plans out there. Could Opus one shot most of those same queries in like 1/4 time? Yep, but that's not the point for me.
DeepSeek Flash for parent then DeepSeek Flash and GLM 5.3 flash as the debating subagents gives best overall results in my experience.
For context I do equal parts sysadmin/devops and having them write software
What harness are you using to organize them? How are they kept from losing context as the session grows?
I run with DSH, I've found it crazy customisable with the Cordis framework and I feel like I understand how the components in it work. So I'm actually able to articulate to the agent what I want for the customisations. I Tried pi and OMP but I couldn't get exactly what I wanted.
As for the session maintenance, I have it configured to trigger a compaction when it ends a 'debate' with >25% of context used but before the compaction it puts what it thinks are key points into an MD file. The approach has worked so far in terms of stopping the agent from getting silly with too much context and retaining information. These flash agents get too flakey for my liking past 25%
DSH is heavily underrated. Have you written your own plugins or something?
I wrote the prompt to get the agent to write the plugins for me lol
DSH has been fantastic, the customisation is great, like I mentioned before. What has got me to stick with it though is the way it presents information like diffs, subagents, tool calls, thinking and sessions. It's nothing revolutionary but everything is where I want it to be and provides just enough information
Without knowing which dots are which, this is useless.
Let’s say I want to run qwen 27B or flash next on a machine I own, off solar panels that are also sunk cost.
How did they price the marginal cost of a token? Depreciation on the box + reverse billing opportunity cost with the power company?
Luna is $20/month. That's cheaper than depreciation of any hardware.
Not that it actually matters here, but my last generation flagship gpu is currently listed on newegg for exactly twice as much as the launch MSRP that I paid. Even decent-enough/established ebay listings for used ones are +$500 lol
Come to think of it, my Qwen box has gone up in value at a rate of $235 / month since I bought it. I guess I could sell it + it would retroactively pay for my Anthropic subscription.
This article isn’t about that. It’s about comparing hosted models.
IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve.
Except it’s very difficult for home users to afford the GPUs needed to run them with acceptable performance.
I have real world data here. On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context.
This is totally suitable (speed wise) for coding.
This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago).
How do you even fit that model in there? What quant is it? Are you using llama.cpp?
No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token.
Seems very slow, but it's actually fine.
P.S. Sorry, where you read “models” I meant experts.
Given that costs for "consumer-ish" GPUs are rapidly rising, I wonder if you'd also have to distinguish two "tiers" of local inference by now: Cost if you bought the hardware a year ago and therefore already own it vs cost if you have to buy the hardware new.
The article notes that this is mainly because the cheap models tend to be distillations of the frontier closed source models, and the developers of the original closed source frontier models have an edge in training their cheaper versions (since they have the source). I'd add that they also have funding, and want to blitzscale. OpenAI / Google can offer a cheap model to get more training data (plan dependent), or have a lot of capacity due to training requirements.
It is also evidence of commodicisation / brutal competition. I wonder if they cut costs because there is stiff competition from open weight models (and arguably we are in the blitzscaling phase where they try to corner the market, not the enshitification stage).
While the distillation argument is brought up again and again, I find it very dubious. This is not to say that it doesn't happen, but I have my doubts that it is a limitation. DeepSeek V4.1 Flash, for example, can reason itself through some pretty complex mathematical problems. Definitely not millenium-prize level problems and frontier models are superior for research-level mathematics, but it is still surprisingly close even for some of that [1, 2].
Where it differs from the larger models is that it has less knowledge; respectably as it performs for its size, 552B parameters are still a lot less than the multiple trillions that frontier models have. Sol/Astra/Opus/Fable may know the solution to the problem directly or to a very adjacent problem, while DeepSeek has to work its way through it from first principles and then sometimes fails (because LLM reasoning is really a heuristic-guided search). But this is also a failure mode that I have seen with Luna.
[1] As evidenced e.g. by ErdosBench. https://www.ulam.ai/erdosbench
[2] Using this as an example over software engineering tasks here, because it tests reasoning capability more than knowledge retrieval.
gpt-6.1-terra has some pretty crazy economics.
I don't think I am going to try to beat them at this game anymore. I've got a machine in my closet that is both collecting dust and accruing value. A very maddening state of affairs.
At this point, trying to beat a major provider is like running your whole house off grid when the alternative is 12c/kWh that is already wired to the premises. You might win if you make it your only job for years on end and have a very specific lifestyle that is tolerant to weird tail risks (assuming you aren't willing to go back to the grid in an emergency).
... and yet ... with codex on my subscription it gets nowhere. Clearing out the entire monthly quota in half a week happens every time now. Didn't use to.
So what's the point? After the codex <> web quota merge next month it'll be useless ...
This is so wrong. I can tell that the poster never used deepseek 4.1 flash for substantial tasks. To me it's absurd how much more people pay for Claude & Co. for even complex software engineering tasks (not talking about finding the cure to cancer here). Just look at the caching prices and expiration times... Did you know that deepseek caches are alive for many hours usually between 1 and 3 days(!)? For a fraction of the price of openai or anthropic?
I'm using Claude all day at work and deepseek privately for very similar tasks. The price/performance difference is extreme.
Is the current frontier pricing sustainable? Or are they bleeding money prior to IPOs?
It's getting cheaper and cheaper (intelligence/cost ratio) actually.
[dead]
Even as the article points out, this fluctuates constantly. In a few months, we're guaranteed to get a new batch of open models, and the balance will shift again.
What's more important is that open models can be inspected, they can be customized, and you don't have to worry that a company will decide that you're not allowed to use them for stuff like reverse engineering things or do security work.
That's why open models, and especially ones you can run locally will always be important. When you're renting closed models, you're basically a serf who has no say in how the model can be used.
Oh really? I have tasks that DeepSeek 4.1 Flash is absolutely murdering everything else on price–performance.
Not to mention the insane discounts I'm getting with subscriptions on it
I'm running full swarming on a 90$ subscription lol
Which sub(s) do you use?
Synthetic, Neural watt has been great too
Yeah, I'd say I get about 10x net of equivalent quality output/usage with DeepSeek over an anthropic sub.
Maybe that’s in the 3% mentioned in the article.
... because they are subsidised in the short term - the business model is to stop you using cheaper & better open models in the long term.
Frontier is a hilariously overloaded term these days.
That's to be expected since all the proprietary models were just released. Give it six months and the Chinese labs will distill from the proprietary models like they did last time.
Yes, but then at that point the newest proprietary models will dominate the price-performance frontier.
The article itself goes against this intuition, open models have taken up as much as 40-50% of frontier for capable/affordable. Right now is very much a low point. The fact there's a lot of churn as new models are released is not unexpected, I think.
If required, so far I'm not impressed with newest proprietary. i have a claude sub mostly rotting at this point, so i come back from time to time until the year long sub expires, but at this stage i will not renew and see no point in doing so.
Have you tried Opus 5.5 and Fable 5.1? They're still better than any of the available open-weights models. (And rumors are Fable 5.5 is coming out tomorrow or next week.)
I have a Claude $200 (Ultra these days) plan and use Opus 5.5 for work with ultracode workflows and all the rest every day. I also use DeepSeek V4.1 Flash 1107 through a separate subscription plan with Pi.
Using DeepSeek I've made the Wayland compositor I use every day. It's finished now in the sense that it doesn't crash or have any bugs, nor can I think of any features to add or anything to change.
When somebody else is paying, I'll absolutely just throw Opus 5.5 at it. Who cares. It gives the best chances that the output is what I want, at no extra cost to me. But for my personal projects, using DeepSeek in Pi with a workflow extension is competent.
Sorry for throwing lots of text at you. All of this to say there's more to it than just which model is more intelligent. There's also speed, cost, and "good enough". Perhaps the problems I'm working on right now are just not complicated enough for "Opus is better" to be a real selling point.
Have used 5.5, ran usual battery of tests, and the downhill death spiral in a science/chemistry context continues. Code wise, it is better than open weight in several ways, but you really have to work hard to fight it from creating such a heavily abstracted mess it becomes unmaintainable for humans. Requires too much handholding for me. It's scientific reasoning is more confident as well, which would be great if it were also more correct.
Open weight models are generally more humble when working at fronteir, seek human input more reliably on thi gs they have no buisness deciding on, and can be trusted to not fuck up codebase in some way. Also, the way claude talks by default and tends towards even with all the "use this standard or that" tricks, grinds my gears in a way that drive me postal.
So, all in all, the shit prose, the constant oversight robbing me of attention better spent elsewhere, and high cost, and seemingly correct and difficult to catch errors on first pass make avoidance/non-reliance a nobrainer.
When dumb models are right, they're right, because likely its a solved problem elsewhere, and when they're wrong its quite transparent. Claude and fable are extremely good at spinning resilliant bullshit that can take quite some mental labor to verify and disprove, usual problem is flawed or deeply i correct assumptions on top of which then relativly sound reasoning and evidence is layered, or otgerwise weird contradictory decisons made "because i thought it would be better this way" that waste 48 hours of gpu time on an invalid model because claude fucked the one small part entrusted to it, which was extremely well defined and reasoned/backed by literature and it just made an executive decision to deviate without consulting or warning about it.
so fuck claude.
I was seriously underwhelmed by the previous generation, reminded me of the hype around gpt2 being too dangerous to live from the same people.
At least Gemini 4 Argon has made writing the front ends I hate and don't care about a breeze.
I have no idea how much that's worth to other people, but it's more than worth the account or three I use.
IMO there's no point commenting on it unless you've actually tested using Opus 5.5. You can't assess the status quo based on even one generation prior.
What is even the point of measuring this if these companies aren't profitable anyway? The prices might as well be arbitrary.
How is this fact related to my decision about what models to use week to week?
It’s not, of course; you’re going to act on your own requirements as you should.
But I think it becomes a little more relevant when making a price-performance comparison such as the article makes between “open” models and “closed” models.
Because the reader is invited by the term “price-performance” to think about efficiency, when in reality the results are being skewed by the fact that the user is not the only one paying for the service: investors are chipping in fairly directly for every token, at inference time.
Users are going to continue to do their own thing for their own reasons, and they should; that’s the market at work. But this is not an efficient market where meaningful comparisons can be made; all of the major closed providers are loss-leading.
Opus 5.5 is currently cheaper than Deepseek V4.1 flash (with APIfied subs, but who cares, that's the price for individuals & startups, unsure why everybody seems obsessed about "API is the price" while it's genuinely not except for Enterprises specifically) - It's quite hard to justify switching outside of the privacy factor.
For us who spends more than $150K tokens a month with less than $5K spending in subs, it's really hard to justify the switch and it's even senseless (for now).
DeepSeek also has APIfied subs, and they have competition in-between so customers can have a great price.
I had a check on artificialanalysis.ai and it shows a 16% reduction in token and 200% increase in time for Opus 5.5 Max compared to DeepSeek V4.1 Flash Max. I don't think the data is very accurate for my real life workload, but for any workload it represents I would think Opus having little advantage.
For general use-case, closed-weight frontier models dominate.
Open-weights excel primarily at coding.
True but let's think about whether this actually matters in practice going forward, given that today, Deepseek/any OS-27B variant can do all the things that most devs need it to.
Let's scale this out 3 years into the future: - Opus 6.7 costs an arm and a leg to run for 2 prompts - Astra 7.89 literally requires you to buy-now-pay-later because it's so expensive - As a bootstrapped dev, your only option is serving local Qwen-4.04-27B that runs on your old RTX (that you bought in 2024)
Sure, it's worse performing. But if you can't afford the best ones in the first place, what's so bad about having a "pretty decent" model that you run essentially free (only paying electricity) if it does what you need it to?
... and if we're all in the permanent underclass anyway, what does it matter that your local model is dumber than the SOTAs? At least you have one! Some people who can't afford GPUs still have to type emails by hand.
(jk, this is actually a cool dataset though ^)
Lol okay
[flagged]
Despite its simplicity, price-performance is not the only metric; this obsession is a typical consequence of extreme overextension of the Friedman doctrine, and it is why so much software and hardware has been qualitatively shitty even when it matters that it’s not. That something can be measured doesn’t automatically make it preferable as a decision-making basis over intangibles.
It’s pretty obvious that local AI has a higher TCO if you look only at money. But there are other things to be considered.
[dead]
Cool story bro