I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.
The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.
Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.
If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.
Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
Cost per tokens is as valid as price per unit volume of fuel.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
That’s not a good analogy because a gallon of gasoline has a known amount of energy in it. The efficiency of each vehicle is also known, at least in a way that is easy to compare on a relative basis.
I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.
Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon.
Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.
Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.
They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.
I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.
I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective.
I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.
Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.
I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost.
If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.
Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks…
It might vary between tasks though. A model that’s great at abstract reasoning might be great at writing math proofs but struggle to write software in <insert language>.
I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
> may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Well that's the problem with these black boxes. You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume "the cheaper and older model is probably going to cost less overall to use than the newer, more expensive one."
I'm definitely using more credit per task, using Fable, but I'm also giving it much more difficult tasks. You're right, it's very hard to tell. I think that using Opus would be more expensive because of the number of iterations I would need to go through.
That's why you have to reframe in terms of total cost per task and factor in model token generation quantity and multiply that by the base cost of the model. Then factor in your time value if you dare. Then you should get a more meaningful business metric.
In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.
I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for
Efficiency is the next frontier in LLMs, and I'm not confident the American companies are taking it seriously enough. DeepSeek, even in a naive API-calling loop, serves something like 80-90% cached tokens at an absurdly low price per token. Using an agent harness tuned specifically for their caching (Reasonix) pushes the cached tokens to 97-99%. DeepSeek is consistently among the cheapest models per task in my benchmarks, while also performing quite well. I'm still almost always using Claude for work, but for side projects, small stuff, etc. and anything better served by an API rather than starting up Claude Code (or `claude -p`) I'm using DeepSeek pretty often.
Anthropic models also shut down on a lot of security-related work, which is what I've been spending a lot of time on lately. I expected Fable to refuse this kind of task, but even Opus 4.8 refuses to build a verification harness for security bugs, as that involves exercising a discovered bug to prove it's been fixed in an automated red/green way, which looks like exploit creation to Opus' guardrails. So, I have to use other models for that work, now, though most of the original benchmarks I built were built with Claude.
Sonnet 5 makes more sense when you pretend the higher thinking efforts don't exist. (His test was on xhigh)
Anthropic's own release announcement mentioned that it's less cost competitive per task than Opus at higher thinking levels. It's significantly cheaper at lower levels though.
I'm wondering if this is going to be a universal pattern of smaller models: they're less smart, so to achieve the same benchmark results they have to think a lot more and hence become expensive.
Benchmarks force models to solve the problem entirely by themselves, requiring thinking. But if you pair them with a smart model (who thinks and solves beforehand) they won't need to solve the hard parts and can run on low/med. I suspect that was Anthropic's intention.
Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
Tokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Concrete example: I’ve been trying to use Claude to generate all my commit messages, but it takes 5-10x longer than if I just write them myself. Mine are less detailed, but one line changes are sometimes inconsequential (especially white space reformatting). I wish there was a model that understood the codebase well enough to generate commit messages in half the time.
The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.
Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.
Given the capability of fable and the shockingly repetitive silly mistakes they made when publishing/updating something, I am starting to wondering whether Anthrophic can afford Fable for everything themselves.
You're paying for a service with known flaws. They do not guarantee correct answers.
Also, LLMs don't "make mistakes". They don't think or act. Every single thing they output is a hallucination. It just happens that the vast majority of things align with reality.
I believe the future lies somewhere in between. I'm working on a hybrid application to reduce our company's token consumption. It runs on our data center's computing infrastructure and on laptops in our community. You might be interested; you can check out the code if you're interested: https://github.com/vfalbor/hibrid
Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash).
Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
Yup. I’ve been evaluating several on openrouter and find token cost meaningless for my work. I haven’t found a great alternative, though the “cost per task” he uses makes some sense.
An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream.
Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens.
The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.
I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.
tool use is another factor, every time the agent uses a tool the entire context is priced at cache rate on top. the same happens when it asks you for input.
Another thing I noticed is the llms perform efficiently/effectively only under the optimum circumstances. May be this just a Claude issue, but when the session goes on for very long the effectiveness drops drastically and I start getting bad answers. This is especially true for design and debugging. Wonder how that ladders up to the token usage
Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.
My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoints? Or just stop using PowerPoints at all. Do we throw LLMs at the mess of wikis and word docs, do we pretend that the policies no-one has ever read actually are how the LLM should think or is it how the work actually gets done - barely documented
The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then
People don't like to hear this but the open models just aren't good for end to end agentic workflows.
There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.
>> just aren't good for end to end agentic workflows.
This is (apparently) the conceit of SteveYegge / GasTown - no model can cope unassisted so chunk it up, run it and if it falls over remember the exact place and restart, merging it all in
But that’s not my point.
I believe that software is a new form of literacy and just as all
Companies and societies are literate now, in the future (tm) companies will run exclusively on software - AI developed software and those who go all out will have the sort of advantages the Catholic Church had over .. guilds?
Anyhow, that’s me being AI optimist. But writing the code is going to be a small part of that transition - almost everything to do with LLMs that is claimed amazing (Computer vision is something else) - almost everything people say we need an LLM is stuff you could have done three years ago but your internal politics just would not let you. Oh look we can see if our policies are being met (you could have written the policies in code and solved the whole problem)
Im struggling to get it out but - almost everything AI is proposed for is stuff a well run engineering firm coukd have taken on. A software literate firm could have done without AI is where firms are hoping AI will
Get them
Imagine how far ahead real software literate firms will be - as long as they don’t burn their runway in tokens
Which is why, the right play imo is still buy in-house as much as possible, engineer around the problems and explore the phase space at marginal
Cost.
Have you been using those models? I've been using a hand-rolled orchestrator with Mimo v2.5 (I seem to be paying $0.017 per million/tokens after their heavy caching) and it's been very impressive. I started with it in Opencode as a harness, then had it build its own micro-harness with stdlib-only Python, then used that to build a local stdlib-only Orchestrator with CLI and web harness, and now I'm using that for improving itself and now multi-project wider-ranging software. I talk to a steward who investigates and plans, then the plans are handed off to parallel worker agents who go through a work, test, interrogate, review, eval state machine for quality (all autonomously) with me at the end just reviewing the work or getting notified if the work items aren't progressing due to the workers getting stuck. So far the only "getting stuck" has been bugs/configs on my part, all at a pretty great quality bar, and at a price that makes me laugh at things like Opus.
I'm still using Claude at work (they're the only approved provider), but wow are the smaller models starting to SMOKE the big ones. At this point, all I'd consider paying out of my own pocket for is the lowest-limit Anthropic/GPT plan to get a big model as the Steward, but I wouldn't pay for ANY of the Anthropic models as the workers who do all the work. And as time passes, I don't know if I'd even do that; the open models are serving SO well.
Yes, it's a "cloud provider" but it's a cloud provider running an open model you can download (and that other cloud providers do host). I just happen to not have a computer big enough to host it.
As for the Orchestrator, it's pretty simple. In essence, it's like "Jira/Trello/Kanban on autopilot". Work items have states, a state machine defines how those work items transition between states, states are todo, in progress, retrying, reviewing, code reviewing, done. work items also have connections, allowing the LLMs to specify a dependency graph, and the dependency graph informs the dispatch order/parallelism, as well as when branches have to be merged. I talk to the steward, the steward has tool calls for interacting with all the data, and the orchestrator auto-dispatches all the work that comes in. I can generate work as fast as I can describe it to the steward, and that's usually the bottleneck.
So far I haven't had to deal with "how do you get the LLM to re-organize the work mid flight due to a worker finding something not accounted for by the planning", but I assume it'll come soon. The most complicated digraph I've tossed at it was 9 items and 4 layers deep. The kind of work I've given it hasn't been scoped large enough yet, so we'll see how it tackles that.
There are probably 100 competing versions what this phrase might encapsulate. Could you elaborate more on which version you are using exactly?
My experience is that frontier models are only marginally better and not close to the cost/value of the open models which are anywhere from 10-100x cheaper.
Perhaps I'm not doing "end to end agentic workflows?"
> People don't like to hear this but the open models just aren't good.
Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills.
That's why most people who are using (excellent btw) open-weight models are just renting compute online.
All of the smaller models from anyone are distilled from larger ones. I assume you are just trying to disparage the Chinese models, but what you are actually saying is that people should only be using the largest non-distilled models, not smaller ones like Sonnet. I assume the upcoming Opus 5 will be distilled from Fable 5.
I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.
The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.
Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.
If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.
Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
Your strawmen are so incredible it's hard for me to believe you've even used one of these coding tools before.
Careful you don't "it's just a text predictor" yourself into unemployment .
>Careful you don't "it's just a text predictor" yourself into unemployment .
says more than anything else could
One more ~~lane~~ layer of LLMs is sure to solve all our problems
> it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on
This downplays the incredible things that can be done with it.
There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3).
Disregard the signal at your own expense.
What incredible things can be done with it?
Cost per tokens is as valid as price per unit volume of fuel.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
That’s not a good analogy because a gallon of gasoline has a known amount of energy in it. The efficiency of each vehicle is also known, at least in a way that is easy to compare on a relative basis.
I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.
Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon.
Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.
Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.
That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.
They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.
I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.
Your comment makes sense but I'm pretty sure yreg is saying the opposite of that—that their task is harder than the benchmark currently is.
(see their follow-up reply: "The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.")
In either case, you need the right benchmark for the right task
As always, the relevance of any given benchmark depends on how similar what it’s testing is to your workload.
Sisyphus doesn't care about energy per meter to move the rock.
Isn't the benchmark working exactly how it should in that case?
I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective.
I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.
Routing tasks to models by complexity like a job for a LLM.
I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.
Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.
I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost.
If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.
Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks…
It might vary between tasks though. A model that’s great at abstract reasoning might be great at writing math proofs but struggle to write software in <insert language>.
I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
> may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Well that's the problem with these black boxes. You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume "the cheaper and older model is probably going to cost less overall to use than the newer, more expensive one."
I'm definitely using more credit per task, using Fable, but I'm also giving it much more difficult tasks. You're right, it's very hard to tell. I think that using Opus would be more expensive because of the number of iterations I would need to go through.
What company is it, so I can compete with them? ;)
That's why you have to reframe in terms of total cost per task and factor in model token generation quantity and multiply that by the base cost of the model. Then factor in your time value if you dare. Then you should get a more meaningful business metric.
In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.
Yeah definitely. I've recently commented on that: https://news.ycombinator.com/item?id=48557890
I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for
Efficiency is the next frontier in LLMs, and I'm not confident the American companies are taking it seriously enough. DeepSeek, even in a naive API-calling loop, serves something like 80-90% cached tokens at an absurdly low price per token. Using an agent harness tuned specifically for their caching (Reasonix) pushes the cached tokens to 97-99%. DeepSeek is consistently among the cheapest models per task in my benchmarks, while also performing quite well. I'm still almost always using Claude for work, but for side projects, small stuff, etc. and anything better served by an API rather than starting up Claude Code (or `claude -p`) I'm using DeepSeek pretty often.
Anthropic models also shut down on a lot of security-related work, which is what I've been spending a lot of time on lately. I expected Fable to refuse this kind of task, but even Opus 4.8 refuses to build a verification harness for security bugs, as that involves exercising a discovered bug to prove it's been fixed in an automated red/green way, which looks like exploit creation to Opus' guardrails. So, I have to use other models for that work, now, though most of the original benchmarks I built were built with Claude.
Sonnet 5 makes more sense when you pretend the higher thinking efforts don't exist. (His test was on xhigh)
Anthropic's own release announcement mentioned that it's less cost competitive per task than Opus at higher thinking levels. It's significantly cheaper at lower levels though.
I'm wondering if this is going to be a universal pattern of smaller models: they're less smart, so to achieve the same benchmark results they have to think a lot more and hence become expensive.
Benchmarks force models to solve the problem entirely by themselves, requiring thinking. But if you pair them with a smart model (who thinks and solves beforehand) they won't need to solve the hard parts and can run on low/med. I suspect that was Anthropic's intention.
Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
Tokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Providers amortize the compute across a batch.
If yours is the only request in the batch it will cost them one full pass through the model.
If yours is one of 1024 inputs in the batch the per token cost is 1024x less.
As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanly
We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)
(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)
Concrete example: I’ve been trying to use Claude to generate all my commit messages, but it takes 5-10x longer than if I just write them myself. Mine are less detailed, but one line changes are sometimes inconsequential (especially white space reformatting). I wish there was a model that understood the codebase well enough to generate commit messages in half the time.
It’s not meaningless at all: every query returns usage and I can calculate the cost.
EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance.
EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”
Can you really calculate the cost easily ? Given most of it should be reduced by input caching read (if you don’t want to have a crazy bill)
Yeah you get the usage back and each element has pricing published
The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.
Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.
Given the capability of fable and the shockingly repetitive silly mistakes they made when publishing/updating something, I am starting to wondering whether Anthrophic can afford Fable for everything themselves.
On top of that isn't it strange that if the LLM makes a mistake you're still charged for those tokens?
They're selling "intelligence", automation, etc but if the service doesn't work as expected the user has to pay for that.
You're paying for a service with known flaws. They do not guarantee correct answers.
Also, LLMs don't "make mistakes". They don't think or act. Every single thing they output is a hallucination. It just happens that the vast majority of things align with reality.
If I use electricity to do something stupid, I still have to pay for the electricity. Intelligence is just another utility.
Spend 5 minutes on the marketing pages of any AI company and it's obvious they're not selling electricity, fuel, or even raw compute.
Agreed with the problem, but what's the solution?
I believe the future lies somewhere in between. I'm working on a hybrid application to reduce our company's token consumption. It runs on our data center's computing infrastructure and on laptops in our community. You might be interested; you can check out the code if you're interested: https://github.com/vfalbor/hibrid
Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash).
Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
Yup. I’ve been evaluating several on openrouter and find token cost meaningless for my work. I haven’t found a great alternative, though the “cost per task” he uses makes some sense.
This reminds me of cpu benchmarks vs actually running games and measuring FPS.
Well, not totally meaningless but certainly can be misleading.
An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream.
Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
The variable missing from cost-per-task: which tasks shouldn't be hitting an external API at all.
cost per benchmark task is definitely interesting!
i've always wanted cost per prompt, but even that has too much variation.
The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens.
The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.
I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.
tool use is another factor, every time the agent uses a tool the entire context is priced at cache rate on top. the same happens when it asks you for input.
Another thing I noticed is the llms perform efficiently/effectively only under the optimum circumstances. May be this just a Claude issue, but when the session goes on for very long the effectiveness drops drastically and I start getting bad answers. This is especially true for design and debugging. Wonder how that ladders up to the token usage
Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.
locally, im about 80k every 30 minutes for a project. can run deer flow to pump them tokens.
but yeah, its the 80s LOC metric since quality isnt captured
would be nice to have these benchmarks so they can be run against models like the qwen family, gemini, etc.
My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoints? Or just stop using PowerPoints at all. Do we throw LLMs at the mess of wikis and word docs, do we pretend that the policies no-one has ever read actually are how the LLM should think or is it how the work actually gets done - barely documented
The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then
>bring it in-house
People don't like to hear this but the open models just aren't good for end to end agentic workflows.
There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.
>> just aren't good for end to end agentic workflows.
This is (apparently) the conceit of SteveYegge / GasTown - no model can cope unassisted so chunk it up, run it and if it falls over remember the exact place and restart, merging it all in
But that’s not my point.
I believe that software is a new form of literacy and just as all Companies and societies are literate now, in the future (tm) companies will run exclusively on software - AI developed software and those who go all out will have the sort of advantages the Catholic Church had over .. guilds?
Anyhow, that’s me being AI optimist. But writing the code is going to be a small part of that transition - almost everything to do with LLMs that is claimed amazing (Computer vision is something else) - almost everything people say we need an LLM is stuff you could have done three years ago but your internal politics just would not let you. Oh look we can see if our policies are being met (you could have written the policies in code and solved the whole problem)
Im struggling to get it out but - almost everything AI is proposed for is stuff a well run engineering firm coukd have taken on. A software literate firm could have done without AI is where firms are hoping AI will Get them
Imagine how far ahead real software literate firms will be - as long as they don’t burn their runway in tokens
Which is why, the right play imo is still buy in-house as much as possible, engineer around the problems and explore the phase space at marginal Cost.
Then and only then think frontier models.
Have you been using those models? I've been using a hand-rolled orchestrator with Mimo v2.5 (I seem to be paying $0.017 per million/tokens after their heavy caching) and it's been very impressive. I started with it in Opencode as a harness, then had it build its own micro-harness with stdlib-only Python, then used that to build a local stdlib-only Orchestrator with CLI and web harness, and now I'm using that for improving itself and now multi-project wider-ranging software. I talk to a steward who investigates and plans, then the plans are handed off to parallel worker agents who go through a work, test, interrogate, review, eval state machine for quality (all autonomously) with me at the end just reviewing the work or getting notified if the work items aren't progressing due to the workers getting stuck. So far the only "getting stuck" has been bugs/configs on my part, all at a pretty great quality bar, and at a price that makes me laugh at things like Opus.
I'm still using Claude at work (they're the only approved provider), but wow are the smaller models starting to SMOKE the big ones. At this point, all I'd consider paying out of my own pocket for is the lowest-limit Anthropic/GPT plan to get a big model as the Steward, but I wouldn't pay for ANY of the Anthropic models as the workers who do all the work. And as time passes, I don't know if I'd even do that; the open models are serving SO well.
So you are using a “cloud” provider and at 1c per million tokens …
Love to hear more about how you structure the orchestrator etc
Yes, it's a "cloud provider" but it's a cloud provider running an open model you can download (and that other cloud providers do host). I just happen to not have a computer big enough to host it.
As for the Orchestrator, it's pretty simple. In essence, it's like "Jira/Trello/Kanban on autopilot". Work items have states, a state machine defines how those work items transition between states, states are todo, in progress, retrying, reviewing, code reviewing, done. work items also have connections, allowing the LLMs to specify a dependency graph, and the dependency graph informs the dispatch order/parallelism, as well as when branches have to be merged. I talk to the steward, the steward has tool calls for interacting with all the data, and the orchestrator auto-dispatches all the work that comes in. I can generate work as fast as I can describe it to the steward, and that's usually the bottleneck.
So far I haven't had to deal with "how do you get the LLM to re-organize the work mid flight due to a worker finding something not accounted for by the planning", but I assume it'll come soon. The most complicated digraph I've tossed at it was 9 items and 4 layers deep. The kind of work I've given it hasn't been scoped large enough yet, so we'll see how it tackles that.
>end to end agentic workflows.
There are probably 100 competing versions what this phrase might encapsulate. Could you elaborate more on which version you are using exactly?
My experience is that frontier models are only marginally better and not close to the cost/value of the open models which are anywhere from 10-100x cheaper. Perhaps I'm not doing "end to end agentic workflows?"
> People don't like to hear this but the open models just aren't good.
Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills.
That's why most people who are using (excellent btw) open-weight models are just renting compute online.
They just aren't good at agentic work.
Also risking it all for some distilled models is a recipe for disaster.
All of the smaller models from anyone are distilled from larger ones. I assume you are just trying to disparage the Chinese models, but what you are actually saying is that people should only be using the largest non-distilled models, not smaller ones like Sonnet. I assume the upcoming Opus 5 will be distilled from Fable 5.
GLM 5.2 is the first model that is competing with the frontier, everything before it existed I would totally agree with you.
I disagree, they are that good at agentic work.