I don't have any evidence/study to substantiate this but I have the feeling there's a _massive_ amount of fiddling with the quant/functionality of the models at different stages of the release cycle and account freshness.
Every time a new model drops it feels wildly powerful, then regresses to some sort of generic SOTA level we've been used to over the last 6 months.
I remember my first coding prompt with, I think it was sonnet 4.6, on a fresh account and it was shocking how much it successfully did off a single prompt - I haven't found that with any other model at any other time and I suspect it was because they put me at the top of the priority list to get a "wow" reaction.
Regression after a release is pretty well-agreed upon, but I'm wondering if they secretly have a huge amount in the tank they aren't releasing. Either as a card to play in the future, or because they can't keep up with demand. I've heard anecdotes of people working in these companies saying that the public models aren't even close to their actual capabilities... Anyone have any opinion on this?
So, IMO, they have much better capability internally because they have 'unlimited' usage. I also suspect a lot of 'Fable' is just a shit ton of nested subagents doing necessary backend work with a very solid fine tuned LLM conversing with the user.
I worked recently for a German megacorp, for whatever reason they gave us unlimited access to a bunch of models, up to sonnet 4.5/Gpt o1 etc, so not the best or latest models. And no agentic tools.
So I built a custom Claude Code clone because they exposed the api from their custom 'LLM Chat' clone in the client and accepted my user credentials for access.
Essentially a frankenstein AI harness using various models for their strengths, including using the OpenAI best model for memory context consolidation and compaction + Sonnet as orchestrator, etc. I was able to spawn 20+ agents to do my work and basically automate my entire job at that corp. I think my internal bill was something like 5k euros per month and I was being careful to not abuse it...
It was better than the best you could get from claude at the time (about a year ago now I think), in terms of task performance completion.
So imagine what someone with essentially infinite spend can do with endless fable agents. AKA US government and Anthropic internal.
I'm not releasing a harness as standalone, rather, I am creating custom harnesses internally for 2 different SAAS projects (I'm hoping to release both by the end of the year). One is related to job app/job search, the other is 3d model and 3d cad custom generation from images/text description for jewlery.
Primarily I am doing this to not have to rely on the more expensive SOTA providers and have been tweaking the harness prompts and toolsets as needed + composing agents as tools (like a Jewlery Reviewer/Validator agent getting called on a worker agents work, Engraver agent etc.) and using DeepSeek flash for everything I can.
DeepSeek is so far more than capable for everything I need it to do for both projects.
Also for fun on the side I'm making a buy/sale shopkeeper fps that is using internal harnesses to drive character ai + conversations. Something kind of like 'Probably Stolen' but you actually have to use voice/type everything to a dynamic npc.
The thing that bothers me is the false advertising. The models are announced with specific capabilities and benchmark scores. Then they silently nerf the models after release. Without a public acknowledgement that a production model has been nerfed and what the new benchmarks are, people are assuming the model has the original capabilities. People are choosing to pay for models based on their reported capabilities. Customers trust these models to deliver the reported capability. It feels like false advertising to me when they say one thing and deliver something less silently hoping no one finds out.
If I order a 16oz beer and get a 12oz pour, I'm going to call it out.
I wonder if the questions about the quality in claude code have something to do with it. Like as long as you say it's great, they can take little steps dialing the effort back.
It's kind of crazy they can do this without informing customers. Essentially we like many other knowledge working companies are slowly outsourcing intelligence to a 3rd party that can, at will it seems, alter that intelligence without informing us. Just means we have to invest more in harnesses, guardrails and keep active in open model space.
I think (2) is the important one. Anthropic provides benchmarks with their model releases. If they are making changes that make models perform worse on advertised benchmarks, how can this be legal?
It's a good point. I mean I have no idea if they are actually breaking the law, would assume their tc's would dissolve them from any legal jeopardy but it's still not right.
Is this where independent benchmark companies like artificial analysis could help ?
Google gives a lot of information to its customers, the advertisers, when it changes things that impact them a lot. Not enough for all of them to be happy, but it does private and public communications on big changes.
> they secretly have a huge amount in the tank they aren't releasing
I think it's more how much immediate profit they are willing to forgo today per account age, subscription tier, and usage type. Burning more compute means they make less or nothing (or have to subsidize) vs dialing up increasing amounts of the ever-more aggressive 'optimizations' (aka nerfs) required to get them closer to viable profitability after their IPOs.
They have dozens of 'knobs' which are completely opaque to us yet each indirectly trades worse results for lower cost. On the other side, they MUST show a clear path to HUGE top-line revenue growth and bottom line profit growth from where they are today by min/maxing new account retention AND renewals-after-price-increases and continual nerfs.
We're not using a somewhat variable, load balanced cloud service, we're in a carefully engineered Pavlovian Monte Carlo casino. There are at least 1,000 different Claudes, Sols and Astras at any given moment. Every time we start a session we're spinning a behavioral roulette wheel. In effect, we're the crabs in a pot of water that's being pushed ever-closer to boiling while they're testing to ensure not too many of us jump out per degree increase (they're fine if the less profitable ones jump out). This will inevitably worsen since they are still a long way from justifying their valuations.
This regression in Opus 5.5 started for me on Monday (rolled out Sunday night?). It's very obvious.
I assumed before the IPO the goal would be to prove that the models are fundamental, capable of building deep complex tasks. But more: prove that they have engineers believing in the value of the tool as offered by ANT(could also frame it as "engineers believe the cost is worth it").
If we are having this discussion about nerfs and regressions before the company even IPO'ed, it feels like they are missing the boat on wooing engineers for the long game.
If the sentiment is not "i can't wait for every model release, they just keep building these tools that are so incredible" but instead "I can't believe they nerfed it, I can't wait until something more dependable comes along", we won't see the jump ship right now. But it will happen.
> If we are having this discussion about nerfs and regressions before the company even IPO'ed, it feels like they are missing the boat on wooing engineers for the long game.
The super-linear scaling rate of capability-per-dollar started tapering back toward linear sooner than they hoped. They can no longer afford to subsidize wooing engineers, nor allow today's primary revenue customers to be less than 'Wall Street Expectation' profitable.
It would be nice if Anthropic would be more transparent about these things.
I'll gladly sign up to be a "Experimental Settings" beta tester (with knobs of how extreme I want to go) for a slight discount (or just bill me the same, just gift me a reset every week? idk whatever).
Doing A/B testing in prod on a frontier model that could really mess with the users in really bad ways is a bad idea.
Why would they ever do this instead of just rolling things out like they are now? What is the incentive to give you anything when they can move faster and just get the same result without it?
Because sometimes when they do these things some users are affected for weeks and leave for competitors, I guess they don't care as long as they don't have to be beholden to stock investors, but eventually that too will matter.
Don't you think that this is a calculated trade-off? You can roll out the A/B testing to 2%, or 10%, or 50%. There are statistics on all this. Individual user feedback is irrelevant.
There's direct evidence - OpenAI employees on twitter (tibo) flatly say they are constantly adjusting models after release, and even specifically state "improvements" to efficiency etc for models post-release. They presumably release to maximize benchmarks at the start, then make them "more efficient", and if you want to give them the benefit of the doubt, then presumably more efficient with negligible loss in benchmark scores.
Because the codex harness and the intelligence tests are separate tools, you can end up in situations like this where A/B tests inadvertently break the tool customers actually use, while the tests say everything is awesome.
I bet some engineer got a bonus for “halving token usage” through “efficiency improvements” — announced around the same time — and not realising that they’ve accidentally lobotomised their model in the process.
“Oops.”
Also, it becomes really difficult to unwind errors like this because the “savings” get burned into profit and loss reports, S-1 filings, and so on.
I feel like I am taking crazy pills when reading the other responses, but come on, this seems like normal human psychology when being exposed to a new thing over and over. It feels new, it solves everything, but then over time you get bored with it. Attempting to proxy it with "account freshness" just masks the problem. It's the reason why people tried to quantify it with stuff like https://github.com/ninjahawk/livenerf because the pattern you are referring to is a known bias.
You're telling me HN users have biases just like everyone else and engage in conspiratorial thinking or reverse engineering "rational" justifications for whatever they're feeling in the moment? no way ... /s
I noticed an enormous drop in quality around this time.
Opus 5.5 was significantly crushing every task I gave it for quite some time.
Suddenly, it starts making the type of mistakes that even Sol 5.6 would not make. It became noticeably worse than Astra.
I've grown so completely tired of the labs changing the quality of the model out from under our feet that I CANNOT wait until there's viable alternatives at the frontier level.
I will gladly forgo the best available power for consistency - and I see that coming soon (hopefully with Qwen 4 or shortly after).
DeepSeek v4.1 Flash is not even remotely in the same league as an un-nerfed Astra or Opus 5.5 - despite what microbenchmarks may claim.
It's a great model, especially for the cost - but there's a huge difference.
Same. I'm doing something security related right now and I swear it was smarter than this before. Right now it keeps pushing me to just allow all connections, or leave a password in plain text, or other insane suggestions. Maybe placebo, maybe bad timing though, but it feels ridiculous.
Edit: Just to add some more ridiculousness:
> The last squash stalled because it opened an editor to merge descriptions. I've stopped it. Checking what landed:
I was using jj with Claude for a while now, suddenly it forgot how to squash? It also started doing that stupid thing where it edits files through python scripts so I don't see the diff during manual mode. This is like 3.5 levels of stupid.
If this is in Claude Code, check whether your session got switched to Opus 4.8. After a cyber safeguards refusal it continues on claude-opus-4-8 with a one-line notice. I saw this on ordinary auth-token fixes, in 5 of 60 runs last week.
That is a setting. The first time you get refused it offers to stop or switch to 4.8 and if you choose 4.8 it automatically downgrades silently whenever it happens again. I just shut it down and switch to Codex when this happens.
I would also agree that 5.5 feels dumber the past week. More weird logical problems. it's not as dumb as 5.1 or 4.8 but it feels like it regressed.
The link hasn't been updated since September but anecdotally I had Claude today when asked to write something create a 'to-do' and instructions for me to write a code block.
Another example is suggesting tailscale serve to proxy from one address:port to a different port (so still address:port) when setting up a reverse proxy.
Whether true or not, this kind of sentiment will eventually destroy their business model with users preferring local models instead. When they are good enough.
> I don't have any evidence/study to substantiate this but I have the feeling there's a _massive_ amount of fiddling with the quant/functionality of the models at different stages of the release cycle and account freshness.
Oh good Lord, they stir the pots constantly. Anthropic is hella naughty about this. Someone else on HN said it well, Anthropic has too much of a "researcher" mindset. You cant experiment on human populations tool use the same way you run AI/ML experiments.
I want a stable tool that works consistently, not a hammer that changes mid-swing (credit Mario Zechner for this analogy).
> You cant experiment on human populations tool use the same way you run AI/ML experiments.
Google and Facebook have proven otherwise. Unless you are very, very lucky you are always part of multiple running experiments on their products, and nobody really cares.
If it's mostly indiscernible from baseline that's perfectly fine and allows for incremental improvement. Think infinitesimal changes in light of multiple dimensions. When you have separate development and stable versions there's always the risk of packing too many large changes and no one actually wanting to upgrade (given a choice) because the product is already good enough. Pro and con for both approaches but IMO, you need mostly small changes and occasional big steps when small ones can't get you there.
In a just world this would be considered consumer fraud, or defrauding investors. Things that if you read the law you would imagine that it came with tens or hundreds of millions of dollars in fines and/or jail time for everyone up and down the chain.
If you're selling model capabilities for money, and you change the mobile capabilities, not changing the model designator (name and/or version number) is fraud, full stop.
> A Volkswagen engineer was sentenced today by U.S. District Judge Sean F. Cox of the Eastern District of Michigan to 40 months in federal prison, and two years of supervised release, for his role in a nearly 10-year conspiracy to defraud U.S. regulators and Volkswagen customers
Unlike here where the "phantom bank accounts" fraud was successfully pinned on tellers and branch managers who were all "independently" just trying to meet management goals...
Last night I was working on datalog, but Fable was acting stupid and its responses were LLM babel. After every response, my reaction was "What?". I stopped using it. This morning I wake up to this in my Claude Code emacs session: "Why are you mixing up the datalog stuff in here? you're working on emacs." Claude Code replied, "I did the work the datalog session sent here instead of passing it to you."
I'd venture to say they are about to drop Fable 5.5 or perhaps the next Opus version. Over the weekend, their current models were objectively much slower than usual and subjectively not very bright.
The local Qwen 3.8 27B might not be anywhere near as smart as Claude's models but I never wake up one morning and wonder if I am part of some test model to see if I will accept something somewhat less capable or not trained as well in the tools I am using day to day. Its also not suddenly going to start refusing to test the security of my system or scanning the code for vulnerabilities or continuing to back engineer that old binary the company lost its code for years ago.
It is something I also appreciate with some of the other labs like deepseek who publish the checkpoint they are running on and still sell you the old one for a period of time as well.
Its obnoxiously anticonsumer to be constantly chopping and changing what a model represents on customers who are paying them an awful lot more to be on the frontier models.
That defeats the whole purpose. You have no way of knowing that the version that the provider claims to be serving is the version that they are actually serving
You can always rent a box and serve an open-weight model to yourself remotely. That's probably the only economic solution that halfway makes sense, given hardware prices these days.
Everybody says they pin. Meanwhile, if they can swap in a lower quant and serve more throughput and make more pennies, they do. It's easy to say, "well, KLD is at 5% so come on that's pretty much the same". And then that's the new norm for a bit, and then comes more traffic and they say, well, shit, let's do that again, it's only another 5%...
Majority of people who complain about nerfed models would do the same about local models.
It's the good old: leave a person on uninhabited island with 10 identical photos and after a month they would have found dozens of differences between them.
I’ve been coding for 20 years and like most don’t code anymore. I ship so much code it would take me weeks and months to review.
Does anyone fear that we accelerate so fast and depend on those super smart models, fire humans, that if at any point for any political or economic reasons the IA costs 100 or 1000 more or is shutdown … what will we do about our projects and maintenance of them ?
Will we still have have enough people and time to treat urgent bugs and dig to understand everything and fix ?
I feel it and it makes me anxious some days that, if there is an important bug and I don’t have tokens anymore I’m doomed or I’ll have to pay expensive api price.
I think about Europeans, if not any reason politics play and cut access to US models that we depend on for work, we’ll end up with subpar Mistral ones and that could be very hard for companies no ?
The frontier models are owned by large corporations that still have some accountability, and the engineers there probably see themselves generally as white-hat.
The question is, what happens when frontier models have nefarious ownership?
Like the recent OpenAI "We found 900 linux kernel CVEs", I think we could assume a nefarious frontier model owner would just start exploiting the bugs.
Whatever future we end up in, the likelihood of nefarious frontier models becomes more possible as costs drop.
To your point, as an individual, it would be critical to have your licenses well advertised that you are not responsible for loss associated with use of your code or app. We all bear the cost in this future.
Even if there's zero malice in what these labs do, the reality is the local models are demonstrating that having the Uber Mensch of models isn't actually the "TAM" they forcast. The fact that a lot of people get along with models on consumer hardware scares them because their large models have a very small actual audience. Which means their future is closer to your utility closet than any magnificence of the singularity.
I have built some ridiculously cool stuff with free tier Gemini Pro handling online research/plans and Muse Glimmer 30B handling "fill in the middle" of Gemini plans; this with a single 64GB+24GB 3090, 2023-2024 i5 CPU (bought right before price fuckery)
Am talking about low level Linux kernel interfacing stuff, my own Google Opal like UI generator...still rough but it's machine state to bang into shape not a bridge. It works but needs the refinement common to early development
Imo due to recent job trope developments a lot of devs expect a git repo that feels like a 3-ring binder with their senior thesis and such tucked within
Local models inability to one shot such is being used as cope. Imo these models are more capable than we want to believe
This all reminds of 20 years ago when sysadmins refused to acknowledge the cloud would take off due to privacy concerns of businesses
Look how that went
Data center needs are inflated to try and capture all the business but future devices capable of optimized battery/telemetry aware local inference would satisfy most people
The data centers serve 3 groups; SaaS workers, data center landlords/CEO/shareholders, politicians being paid off by data center landlords/shareholders
The other billions on the planet do not directly benefit as much. Merely as a side effect of trickle down
I spent about a week as a side project trying to get qwen 2.5b running locally to be a job application filler bot with toolset. It was amazing to me that it was able to apply to about 65% of the jobs, based on certain issues that might arise, sucessfully with that toolset, when given the job app urls as the only starting point info (plus user job data/resume in json).
Of course the Sota models on the same custom harness got to about 99%, but I'm amazed at the performance of such a tiny public free model running on my 3080 laptop gpu. If it was actually finetuned for said task I imagine it could get to 90%+ completion.
Right, my point is quite clear: Even if their "total addressable market" is a real number (big if), that TAM is probably easily divided up by tiers of models, and a significant portion of those tiers is already covered by local models.
Of course, any given person might want their own navier-tokes agent swarm of a solution, but the number times you want a bot to fill out a resume rather than solve a millenium problem is absurdly lopsided.
So it's just kinda, ok guys, are you going to just try to hold us hostage with your magicaly super-duper-intelligence, or are you going to admit that having great models means humanity gets a few smart people to be even more useful for whatever they're working on.
The rest of us arn't going to even see that future to begin with, because food, it's not about how much food humanity grows; it's how it's distributed. Unless this AI stuff solves a problem _no one_ has solved for food, it's just going to be another power chair to fight about who gets to sit there, who has the right to sit there, or like I suggest, it's just a utility thats to be managed by the same governance all other utilties do and has the same problem thereof.
Yeah, I'm also amazed how the conversation is always about 'coding', like these models are only good for that, and are going to replace all software engineers while leaving everyone else alone.
IMO these models are best at non-coding tasks. They of course have to use a computer, but from CAD generation, 3d Modeling, emails, legal and academic analysis, etc. these models are incredibly capable, and we are going to see a massive shift in the entire job market due to this, and ironically I suspect one of if not the most protected career as a result of this massive upcoming buildout will actually be software engineers.
Essentially I agree about not reducing them to just coding apps.
Am ultimately whittling everything down to Linux kernel + inference + data model so far captured from my current OS use.
Am ready to be done with syntactic programming of the machine. Rather than label the filesystem with context I can label entities in the viewport and those entities can happen to look like word processors not just video game characters.
An basically making a Blender-clone designed to browse and edit models that replaces the usual desktop metaphor.
Then I can noodle the virtual geometry, label it and understand under the hood its just a function factory mutating geometry of memory address truth table and display truth table.
The last gasp/push to rid ourselves of OOP is upon us; code objects of all kinds gonna be dead sooner than later.
Not sure I agree then software engineers will be insulated when everyone can do state management through visual edits. Maybe visual design work will still be a thing; the how to get machine into desired state is most certainly not going to be with visual syntax structure fir much longer
We do, but it’s not definitive. We are building a battle system on top of OpenCode [1] and Opus 5.5 seems to have dropped since Nov 3rd.
If you calculate Elo only before 11/3 and compare to only after 11/3, the Elo is 52 points lower after. Unfortunately we are still in beta so this isn’t conclusive (confidence interval of -160 to +40).
If we do this calculation on all our models, Opus 5.5 is first in apparent drop. (Again, not definitive, but supports the conclusion.
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I made a CLI entropy generator project.
It's hard to tell, but I think they also pin the harness version? Updates to Claude Code can also have big effects on perf, and Claude Code updates a lot.
This is a good point. Importantly, Claude Code updates change the system prompt and that has both direct and very broad effects on the output. Livenerf hasn't changed the Claude CLI version in 2 months.
They also have this comment in getting started: "Pin the CLI. This is not optional: a Claude Code update changes the harness, and a changed harness looks exactly like a changed model. Turn off auto-updates and write down the version you're pinning"
How do we know that they are not targeting this specific "benchmark" and serving it the same quality model?
> livenerf makes everything else deterministic: frozen prompts
One way to test my theory would be to run the benchmark yourself. (This catches if they are targeting this one account running the benchmark, have to say kind of unlikely...)
The other would be to see if something changes when you include this in your prompt: (this is the system prompt used by this test)
```You are completing a task that is graded automatically and exactly.
Work carefully. Follow the output format in the task precisely: graders do not read anything outside the requested format.```
To control for changes caused by the addition of this extra prompt itself, we could try to add some other piece of text that is semantically the same in meaning. (wont be perfect but maybe its close?)
To isolate impacts further we can do the same test with other non-nerfed models. (we ask, is Opus 5.5 the only model that is suddenly doing a lot better after i add this magic string in my system prompt, or will gpt5.6/quen 3.8 etc get better too)
Exactly right. I feel like anyone with a comprehensive eval suite (companies running their own agents) should see this signal and it would be a well known fact by now if that were the case.
The only thing I could maybe see is that any changes that are being made are happening on the subscription tiers on not the api tiers, but even that seems like quite a stretch.
Like anyone who is building agents with a good eval suite would know, models have been pretty consistent and even small harness changes can impact performance considerably.
At least when I saw this thread I ran subset of my evals and therewasn’t any degradation. But I am using opencode with business copilot so it might be completely independent from average code + Claude subscription experience.
This should be at the top. I'm so tired of people vaguely claiming models are "nerfed" when it's far more likely that your subjective experience has changed...
Definitely would be good to see more objective figures but benchmarks arent always great samples either. This one is also showing its not degraded but IDK methodology https://www.bridgebench.ai/nerf-bench/claude-opus-5-5
If there was a silent degradation then the recent chatter about suspected nerfing is what you would expect to see. Of course that's not evidence, but I dont think we can say it's far more likely that its all subjective.
I mean the basic intuition doesn't seem unfouded. Maybe people can confirm/deny this contention for compute. But Anthropic is compute constrained. Inference takes compute. Model development takes compute. Isn't it plausible, and even likely, that there will be periods of contention? Would Opus 5.5 performance just silently get worse if they gave it less compute, as opposed to just being slower or something? Or do we have information that suggests inference compute is affected by model development?
Monday night was my first session with the nerf and I can't say if the capability has diminished, but absolutely noticeable changes:
- claude used first person pronouns minimally before ("I","my") and suddenly like an average of every 3rd response has them.
- claude introspection'chatter' has increased like 5x over last week ("I'm doing this now. I will run this. I will check that."
- tool calls are like 6x more than what they were. last week it was <prompt>, thinking, tokens, work, response. Now it's prompt, thinking, introspection "i'll do this", tool call, tokens, introspective response, tool call, work, tokens, tool call, ..., response.
I think it's similar to how an artist's eye is ahead of their hand [1]. As you get more experience with the models you start to notice their shortcomings more and more. When a new model comes out, everything seems crazy good for a while, until you start to realize the weak points again. Benchmarks aren't everything, but people have created numerous benchmark trackers because of this phenomena and no one has caught degraded performance yet, other than this post catching that something changed (no connection as of yet to performance though)
Sorry but I have to disagree. Personal experience is subjective, I get it. But also like an artist, you develop a pretty intuitive sense for how things are working. If you do a lot of digital photography, you could immediately spot differences in camera settings just by glancing at the output pictures. Someone who works a lot with Claude Code or Opus 5.5 can reasonably develop a similar intuition.
With Opus 5.5 I (subjectively) observed decreased performance across a variety of projects from greenfield to mature, across a variety of tasks, and a variety of days. Across a variety of projects is evidence against the claim that this is merely a model struggling as the project grows. A variety of tasks suggests it's not just "oh you're doing something you wouldn't have done before" or "your expectations are just higher now". Over multiple days suggests it's not just my subjective mood that is changing.
No one catching degradation in benchmarks doesn't really prove anything besides Goodhardt's law. It could mean that the performance truly hasn't changed, or it could prove that Anthropic runs the same benchmarks when verifying the performance. Just like I can have a fully passing unit test suite while my entire project crashes and burns, the benchmarks are only good to the degree they accurately represent real usage. Which in my experience they don't.
And finally, in this case with Opus 5.5 there are people finding concrete evidence of the degradation by analyzing their Claude Code session history.
So to my main point, I wish people would stop trying to gaslight us or defend Anthropic. No one outside Anthropic truly knows what's going on with these models. We have some direct statements from Anthropic. We have tons of reports online that seem to indicate tons of users (subjectively) experiencing worse performance all around the same time, and we have our personal experiences of whether the models perform the same or not. So rather than assuming everyone with a bad experience just has bad expectations or is some sloppy vibe coder, maybe we can just say "it's unclear what's going on." (Not saying your comment is pejorative though)
What is most appalling is that a significant accidental degradation happened back in March, was explicitly confirmed by Anthropic, and when these conversations happen again, everyone forgets about this important prior incident.
A large group of people reported that Claude Code's capabilities were dramatically reduced. And we heard then what we heard today; we need objectivity, we're being paranoid.
But there was indeed a bug, a bug that excluded thinking tokens from being included in requests after 1 hour idle. Reasonable for an old request one time, but the bug was that thinking tokens continued to be excluded even if you started a /new session, unconditionally.
And now we can't see thinking tokens and look out for this issue.
i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.
Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.
Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?
I've got QFN making a C(sub/superset)->microcode compiler, writing it in OCaml. It's come up with optimizations that Claude didn't come up with (in the 4.6/4.8 era) when it was making the previous C-based implementation. The compiler generates microcode for an algorithmic state machine/transducer in addition to Verilog. I've been working with QFN on this for about 4 days now and it's pretty close to feature complete - yesterday it was able to compile our Game of Life C implementation and the microcode ran on a Tang Nano 9K FPGA board with LCD displaying the cells. I do not think this is an "easier" task.
Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
Local AI is the way forward. Using someone else's harness and someone else's model is just borrowing a workflow that's liable and likely to change on a whim.
100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
I noticed Fable decline in quality a couple weeks ago. And I've noticed Opus drop over the last week, even after bumping to a default of Extra High reasoning for almost everything.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I've been using Fable for a couple months as a relatively independent, long-horizon task executor and it worked well. About two weeks ago it completely stopped working. Like ~85% success rate down to ~10% in my workflows.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changed in terms of difficulty, but it's been kind of shocking.
I have the same exact experience. Working on one project, brainstorming. It was brilliant in July and August, like a seriously smarter, more experienced version of myself, reading in my mind, one step ahead of me, when pointed out a mistake it would remedy on the spot with the exact brilliant solution.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
Same for me, especially the last couple of days. It was really frustrating and it felt like Claude was sick of working with me and just didn’t give a f*ck anymore
I guess there are experimenting with they can enshittify. But even if they undo this current change, all commercials model will eventually be enshittified.
I read somewhere that the test might switch models mid conversation which stops the other from reading the previous model's reasoning. I don't know if that's true, but it sure feels like that quite often in the last days.
I just cancelled my x20 anthropic sub after their last usage reduction shenanigans, and moved over to DeepSeek and opencode go. Yeah opus is a bit smarter, but not even close to justify the 20x price relative to DeepSeek.
I'm a pretty voracious 20x user, and OAI Biz Pro user; however I'm not an engineer by training, a previous tech adjacent dev team leader who always relied on pseudo code and broadening my horizons when dealing with my team. ( AdTech, DTC marketing )
So I'm AI pilled and in another career, and started to think, what if the problem is the inputs? They release Opus X.X and at first it's a rush of their best users, best use cases, upgraded lanes. As weeks go on, adoption increases, cruft uses start to poison the model? Is this how the ongoing evolution works? I'd love to get Sonnet 5.5 and Opus for review in a place where I could reliably daily drive for X months / Y years etc.
According to our measurements this isn't true. We re-ran Opus 5.5 today on the benchmark we ran in September, with the same prompt and settings. Before and after, side by side: https://playcode.io/blog/macbook-svg-benchmark#reruns
Max, streamed (128K cap): 128,000 tokens, $2.56, empty file, in September and today.
Xhigh: 86,096 tokens for $1.72 in September, 107,333 for $2.15 today.
The Batch API bills half price. Max ran out of the 128K ceiling again, to the token. Given 300K, it drew the same kind of laptop with 5% fewer tokens. Xhigh used 25% more and drew the same kind of laptop. That's inside the noise: 17 identical calls at high ranged from 17,757 to 59,705 tokens.
The model is still the best of everything we've tried. Most likely the next best would be Fable 5.2. But we don't like Fable because essentially the result is the same, just the price is higher. Its medium and high are close to what Opus 5.5 Max had, but the price is higher.
(I run Playcode, an AI app builder. We benchmark every model before we add it.)
I noticed the same, and posted about it three days ago.
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
If we're being pedantic, it may be deterministic from the point of view of Anthropic. They can guarantee that my prompt X results in actual prompt X+Y (Y being all the settings they control), which would hopefully result in deterministic output Z. From my point of view, Y may as well be random, resulting in non-deterministic output Z for my input X.
It really though?? Even if you set temperature to 0 I saw research showing that the scheduling of the order of operations in the GPU firmware itself can change trailing decimals places and affect the output token
I'm aware operations can technically finish at different rates throughout all of a GPU's cores, but I am not exactly sure that GPU firmware would change the outcome of a calculation (unless there's something specifically in the firmware for it)
Right lol, i didn't know we'd built a machine that was capable of getting critical level security deterministically correct. That's not me, at least...
You don't need to worry about this. If you've outsourced your thinking to a cloud model, it's going to take your job soon enough. I hear layoffs are popular around the holidays ;)
Multiple instances of failing a Bash tool, calling bash (lowercase) and getting rejected and even reprimanded by the API "No such tool, call Bash instead".
Killing it's own shell, multiple, multiple times in the same session. I'm sure I saw a phrase "I did it again".
It's so shit that it must be damage control. The alternative is to limit usage and that'd affect the IPO even more.
I noticed its been trying to drag out the conversation on every topic. Maxxing token consumption for the dollars instead of staying on point. Makes sense for them but its frustrating for me as a user. I suspect this is going to be the AI version of enshittification. Now its only missing some badly needed in-stream ads.
It blows my mind how people still build their workflows around depending on services like these, with how easy it is to just self host and forget about all of this. You can buy a GPU that can run Qwen 3.8 flash next for about 9 months worth of the highest tier Claude Code subscription. You'll never have to worry about token limits, the model changing out from under you, getting banned, etc.
This really depends on your infrastructure. In this case it all depends on your hardware, and there is no secret sauce, you can buy the same GPUs or better than the big players are using. Self-hostef GLM in our company is faster than any commercial model. Of course the hardware was not cheap, but completely manageable for a medium or large company.
Speed can mean two things: rate of token generation or time to finish a task. What maters it the later and a model that takes a faster path to the result and uses fewer tokens because of that comes out ahead in practice.
Superior? Sure. But do people need the capabilities beyond what can be reasonably achieved with local models? I think the people that actually sit in that gap are vanishingly small, and if your title contains "Engineer" or "Developer" or something like that, you are not one of those people (at least for doing your job)
As for faster, maybe? It depends on the metrics and what you're doing with it. If you mean TPS, no, I get much higher TPS and even faster TTFT on my local setup than frontier models. If you mean time to complete tasks... maybe. If you're throwing whole projects at it to vibecode, yeah, a frontier model will dump slop so fast it'll make your head spin. If you're doing actual methodical development, like doing spec-driven development and actually reading the specs before spinning off agents to implement them, rather than just having another agent review it and yolo, you'll find that frontier models don't really push the needle that much, and you'll be making high-quality projects.
>> But do people need the capabilities beyond what can be reasonably achieved with local models?
Yes? If you're building anything reasonably complex, it is a ton of work to keep everything in your head so that different parts of the system work seamlessly and work you do in one area does not adversely affect another area.
Is there a model that doesn't need babysitting? Half my day is spent untangling BS that Claude and Codex hallucinate. Particularly around hallucinated product requirements. Its still faster than hand-writing code but only just. I'm seriously considering just going back to the early copilot days where I lead coding and just let it try to auto-correct. That was a great balance of AI enhancing my speed and workflows without removing any confidence in what was being shipped.
give your models a more organized context/harness if they are hallucinating product requirements... and I'm not talking about some hulking context framework. just organize your documentation in a canonical way. This product requirements hallucination is something that would bite me constantly if...they ever actually hallucinated them (and they don't, ever). This is most certainly user error.
All the time. All the frontier models love to take a requirement and bend it into a need to create an obscene amount of overhead. I needed to add a permissions for a user type. Instead of adding the type to the existing list of permitted types it created a bespoke experience for that user type complete with a full testing suite. This is without diving into the absolute insane amount of no signal tests that these things generate. It once decided the privacy policy should have its own testing suite, which basically amounted to 50 different regex tests looking for "key" words and phrases, and snuck that into a PR for something tangentially related. They love to burn tokens on creating useless shit and it happens on the daily.
We are literally in this thread because so-called frontier SOTA Claude models are shitting the bed and require babysitting. So this argument is essentially moot.
Qwen 3.8 (and maybe soon 4) series models are probably the best right now for local AI. Specifically, for lower end hardware like 8gb GPUs you might find a lot of use in a Qwen 3.8 variant like Underdog Saluki 27B.
Today this is absolutely true. Local models have gotten very good, but they aren't there yet. The cost of a Max subscription or even API pricing is overwhelmingly worth it for most users, even if you ignore the enormous capital costs of running the competitive models locally.
But the in a million years bit, even taken as rhetorical bombast, is of course silly. The difference between local models and the frontier models is ever shrinking. Extrapolating out and eventually, sooner rather than later, the difference is going to be a wash.
Unless all of the vendors stop progressing open local models, the future is clearly there. It's why I think AI is important and valuable, but also the current leaders have a dubious financial path forward.
In a workplace environment I don't think "cheaper" is necessarily the thing to optimize for. Subscription models save you a lot of time (and money) by getting things right faster, solving bigger problems, etc. A cheaper model just consumes more human time.
But with a local model with essentially free tokens, I can set it to work on tasks in the background without having to worry about any usage limits. Deferring simple bug fix tickets or other low complexity to an essentially unattended local model works just fine. And if the result occasionally fails review, it still didn't burn paid usage credits.
A local model tops 60 tokens/s max. Running it on the background means not being able to use on the task you are working on. When working on local hardware you cannot do concurrent requests. In deepseek for example you can have 100 agents running each getting at 120 tokens/s easily.
In local models you are limited by your own infrastructure.
I do concurrent requests just fine and they have no noticeable performance impact. Not all local hardware is equal. Also, humans have to sleep. Agents don't, so processing stuff over night and reviewing in the morning just gets more utilization out of what is already there.
With cloud agents I also process stuff overnight. But my gpu is limited. If I want to OCR a high number of documents or vectorize data, I can't run an LLM at the same time. I use remote models and use my compute for running commands on my desktop machine.
I sure do. And 2-4 years in the future when this insanity is over I will buy and own local llm hardware. But certainly not now with those prices and speed of progress that obsoletes everything in a year or two.
I made some assumptions and was foolish. Sadly not a new tale. But still in context I do think for every You there are several other Not Yous who are planning infrastructure who aren’t keeping an eye on the local escape hatch.
Compared to Opus local models Opus 5.5 is far superior, also for something like a dgx spark at 5k that would be 25 months ignoring power costs if you chose the $200 plan.
And that’s looking at a large model, to host the equivalent sized model as Opus 5.5 you’d need more than a single dgx spark. A fairer comparison would be to Open models hosted remotely or comparable weak models like Luna which you would get away with a $100 plan or less. That also doesn’t account for token prices continuing to go down for the same level of intelligence.
Qwen 3.8 Flash Next is 125B with 6B active, which with quantizing and system memory offloading (strata) runs nicely on 32 GB cards (I have actually run it on as low as a 16 GB GPU and 16 GB of system ram, but I don't recommend it for serious work). I was running it on a single r9700, but now I'm using 2 r9700s because I'm splitting my server with a friend, and they bought one as well.
On the single r9700 I was able to run q2 with 4 slots of 256k, and I think it was about 40 - 50 GB of system memory offloading? Got about 60 - 80 TPS per slot depending on concurrent use. With the two r9700s we are running q3 with 4 slots of 256k, using about 100 GB of system memory offloading, with mtp enabled. Getting about 70 - 100 TPS now.
i dont understand how people keep comparing local models with frontier models, they dont belong in the same discussion especially for coding usecases. they solve different problems under different constraints.
I don't understand what do you mean. They both solve the same problem. Some models solve it better, some worse. And in some cases you can only use local models, so they solve more problems.
"Frontier" (closed) models are better by definition, but if frontier models were good enough for you half a year ago, open models are good enough now.
What is the alternative? Local models? I do not think so. Maybe Kimi K3 (still not local) as this model can be served by multiple providers? However Kimi K3 sits between Sonnet/ Sol and Astra/Opus so still not the same level.
No. That is just Anthropic allowing US based companies to host it in the US and EU. You can buy a few B300 and run Kimi in your data center as a company. Not possible with frontier models from OpenAI and Anthropic.
I can buy a GPU that can run Qwen 3.8 Flash Next for 9 months worth of the highest tier Claude Code subscription, but then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
> then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
By choice, maybe. New models release faster than every nine months. And, evidently, you won't be running nine months of some fixed meaning of "Opus 5.5". What that means changes over time, out of your control, and without ever even being officially told.
20 euro claude sub is too good compared to a 6-10k hardware purchase to get something thats not even as good. Once the enshittification starts on full force it will be a different situation.
I honestly thought usage was going up. Now, I dont use LLMs nearly as much as a lot of people here, but I used to hit my limit very easily, and now I haven't hit it in wmquite some time.
I don't know how you thought that when real usage is down 10x since 2 months ago, they publicly announced announced several usage decreases and related issues, and its a constant point of discussion/criticism online.
i haven’t heard anyone say they downgraded their car to the children’s toy power wheels and regretted it. all the same, i’m confident that it’s a bad idea
Problem is that everyone loves to pretend their CRUD SaaS is special and requires the best of everything, so they 'need' the absolute frontier with as many tok/s as money can buy.
R9700. I walked into Microcenter on Hallow and picked one up for $1900.
I was also using a 7900 XTX before that which was reasonably capable for single user usage, but I share my server with a friend, and the 9700 is a much better fit. Also nice for doing sub agent workflows that I couldn't really do before.
> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
This has happened to consistently across OpenAI and Anthropic models that I can't believe people are still denying it. Obviously they're silently quantising their models or making other efficiency changes. If they can save 40% RAM and only hurt intelligence by a few percent in their benchmarks, they're going to do that. Unfortunately their benchmarks are obviously not representative of the real world, and these downgrades are very noticeable for people who rely on these models for work.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
I guess the question really is: is this malice (trying to trick people into thinking new model is better by nerfing old one), or is this capacity constraints, where they obviously need to make room in their infra for inference of the new model by shoving the other models into a smaller box?
People are denying it because it's simply not true and for sure has not happened consistently.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
I've "called" the last 3 out of 4 new model releases. Maybe my account it perma-tagged as one that gets downgraded to re-allocate resources before a release but about 5-7 days before a new model Claude suddenly gets dumber and dumber on executing canned skills that I've used for 6+ months with no issues. The cynic in me thinks they are dumbing down models right before a release so the new model seems much smarter.
It’s also true that people that perform their own evals see wide variance uncorrelated with “nerfing”. Human cognitive bias makes them only see that variance at specific times and then they start projecting.
Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.
It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.
Yes I have similar anecdotal experiences - it gets dumb before a new model. What I imagine in my head is Anthropic using all their GPUs to polish off the next model and they need to borrow GPU cycles so they cut down the thinking.
There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.
There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.
I was being sarcastic. LLM output is not deterministic (at least not in any meaningful way beyond the fact that everything in the universe is deterministic).
Correct, but that's because of added random inputs not because the model itself isn't deterministic. Put in the same random seeds into your model runtime and you'll get the same output every time.
> everything in the universe is deterministic
Again, mostly being pedantic, but not everything is deterministic. Radioactive decay is an example of a phenomena in the universe which is non-deterministic. Most things are, not all things.
Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.
They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.
Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.
My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.
I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.
Order of operations affects the result in floating point, even moreso at lower precision. GPUs are massively parallel devices with many different sources of latency, so order of operations is not guaranteed for things like dot products, where many threads contribute to the result. Consequently, you can get non-deterministic outputs even though nothing else has changed.
Some teams I know started moving all linting and validation into LLMs and markdown files, the reasoning was that they asked agent to review changes that were later found to be buggy and agents flagged these bugs. And that’s great but my immediate worry was that: it finds it now, but will you check every new model for all these bugs? And what if you need to switch providers? Or your provider silently tweaks the model to improve their margins? The answer was “well also build deterministic linters for all these problems”, but you know that’s just wishful thinking - if you can get 99.9% done with just a little bit of fuzzy text and the next 0.1% is a lot of work it won’t happen. So you have model behaviour lock-in that will bite you suddenly when you least expect it.
Or maybe LLMs will just keep getting better and cheaper? It’s all a casino right now.
Every LLM company must keep getting better and cheaper to compete. If you game out them running low on money, none of the outcomes actually result in you spending more or getting worse models, just switching to another one.
If you have 99.9% already - that's better than any pre-LLM deterministic linter.
This makes no sense. Even if you're all-in on LLMs why not just ask the LLMs to interface with the static analyzers. Fix the bugs, add new rules, run the commands you always forget, etc.
It's like saying "save this json as .txt but using valid json syntax." I mean maybe that will work most of the time but WHY?!
- 1 week before releasing new model, they nerf the old. Probably to reserve compute for the new release. (insert your evil corp motive here). This is the third time I have seen this now.
- They run A/B tests continually. Hence the "how is Claude doing: good/okay/bad" question.
- They try to shrink the model while keeping performance, using quants, etc. they keep the name though. Opus at the first, is often not the same Opus a month later.
- My friend works at a top 100 spender on Anthropic. He complained about nerfing to his point-of-contact at Anthropic, and they did some setting change and it didn't happen anymore.
- He, European, says the models can get worse around 4pm, which is when America gets to work
I'm surprised why folks are either all in on frontier subs or self hosting on their own hardware.
To me it seems there should be a very viable middle ground of paying for open model hosting by a trusted provider - unless I'm only hearing from 24/7 vibe coders
You can still control it if you can trivially spin up a node with eight B200s and 2TB of memory to run your Deepseek v4 instance. Nobody on a subsidized plans is doing that. Local models that fit 128GB memory don't come close even to a "nerfed" frontier model.
Actually Qwen3.8-flash-next comes very close to a "nerfed frontier model" the way I host it locally at 4.75bpw with int8 kv. It is pretty much equivalent to opus 4.6 up to 4.8 on coding. What it lacks is some factual knowledge - this is to be expected, it's a far smaller model. But if you run the kind of coding tasks where you have to spec them in high detail anyway (large very custom environment) this model is very usable.
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
I've been very impressed by it and recommended it to everyone. 32gb ram, 12gb 3080ti, and I run 3.8 flash next coder (half the experts ripped out, leaving in coding) int8 kv 128k context at 35-55tk/s and I agree with the opus comparison; I personally say 4.7 is what it seems like the most.
I read stuff like this and just think that if these open models work well enough financially you'd still be better off using them via OpenRouter or some other hoster.
Local is not worth buying hardware for, it's only good to run on the stuff you already own. As soon as you want to do work you are getting screwed almost as hard as people in the 90s did with $2000 pcs that went out of date in under 6 months. Just pay 20 bucks and use the frontier models, they have better harnesses and features than we have access to anyway.
Definitely a significant change. I noticed it in just daily work. Quality of work down, mistakes up, repetitive errors in scripts - rare earlier, now a few every session.
Definitely noticed some anecdotal declines this past week or so. Missing steps it didn't previously, needing reminders of tool calls and skills that it should auto apply.
Lots of "this is a potential issue, I didn't check into it though" which is super frustrating to see. The model just points out major flaws in logic or barriers but goes no further than a comment on it.
I've got a Qwen coder model and a flash next model locally now, so it's not the end of the world. But it does need to be remedied - are we seeing them prep for a new model deploy when this happens? Or are they experimenting with leaner hardware to drive down costs ahead of IPO?
I’ve been experiencing unreliable performance for a while and can’t imagine this is solely due to new model training.
It does seem like Anthropic is experimenting with how much less they can offer users. I also get the feeling they are judging what resources a task deserves and dumbing things down accordingly. I’m seeing lazy responses and skipped steps more frequently for non-technical queries.
It’s definitely possible the average user hasn’t noticed, unfortunately.
I moved from Claude to Codex and am happier. The prose is less grating and results comparable. As much as I don't like Altman I am tired of Anthropic's antics.
I tried this and while the prose is indeed snappier I find the code quality of Codex truly horrific. It's dense and uninterpretable. I almost wonder if they have made it purposefully opaque so that it's harder to resume work without Codex once you're using it agentically. Claude is still much better at writing quality code that is fit for human consumption.
I don't think Astra's code is quite as bad as you say, but yeah while it's much better at non-annoying prose, the code definitely isn't quite as good as Opus 5.5.
I had a baseline evaluation from before Nov 3, so I re-ran it this morning.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
If you paid for X but are now receiving Y, you should be entitled to get your money back.
Every time I consider going back to Claude, I remember that mysterious week in February when Claude was failing to do even the simplest things. Things were so bad that BigPickle was correcting errors Sonnet was making.
And let's not forget the king of A/B testing here. Google has no issues breaking products for months just to test some ridiculous idea.These people should be publicly fired.
> Every time I consider going back to Claude, I remember that mysterious week in February when Claude was failing to do even the simplest things
100% agree. I don't care how many reddit bots i see praising Claude, I bought max once wanting to see what the fuss was about, and the next day they broke the product and it was really embittering
Google is known to make user facing A/B testing whenever they feel like it.
They can one day decide to rearrange the Gmail interface to absurd levels by removing the TO field or replacing everything with a big Gemini button. Then push it out without any internal testing to half of the planet, and suddenly you are doing IT support for grandma who is convinced North Korean spies have hacked into her email...
I guess it's just capacity problems. They probably have a fixed amount of compute and they need to make space for new models that will be released. So they quantize existing models e.g. in half so that can use 50% of current capacity for new models. They have such massive scale that you just cannot dynamically get that amount of new compute online so they need to work with what they have. No provider has thousands of GPUs idle.
There are a couple documented things where it has degraded: in late 2025 or so when they transitioned to TPUs, they had a bug in the compiler affecting sampling logits in a cheaper approximate way than GPUs and that caused a big degradation. They never told users "sometimes we'll sample logits in a cheaper approximate way," but the main issue was the bug and not really an intentional thing at least from how the characterized it.
TPUs only made up a small percentage of the fleet, but, exacerbating that bug, if you got routed to a TPU instance at some point in your chat then another bug kept you routed there for the remainder of the chat.
The second one I'm aware of is earlier in 2026 they decided to save usage by stripping out all thinking tokens if you left your chat idle for one hour (cache timeout) and then came back to it, causing massive model performance degradations. This one also made me very skeptical of Chinese distillation, until the reports of how you could use Haiku to extract the raw reasoning of the larger models.
They fixed that, but recently they switched to doing auto-compactions for long running chats in anticipation of them going stale in cache. I think that one is usually what you want anyway and there is at least a way to recover to the uncompacted chat if you really need it, but it might also be a source of some degradation people are seeing now.
I think the bug one wasn't bad intent, but it was a fact that even without the bug you got different behavior with the approximate logit sampling on TPUs. I say that was jut a minor platform optimization though and wouldn't be too upset about them not disclosing it.
But the harness one was not just a random harness regression; they were getting a huge influx of users and pretty clearly did it to save capacity. Not a great thing when you might be relying on attention to those thinking traces to reduce the risk of prompt injection etc. and it could have gone beyond major task degradation into a safety/security issue.
Yeah, perhaps but they are also trying to make the harnesses more efficient so they are competitive with their competitors on efficiency. Reducing token use can also reduce context size and improve aligned accuracy in general..
Through the lens of a conspiracy theory this could be viewed narrowly as "cheating" their users to reduce load.
But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
> But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
Anthropic publishes extremely detailed changelogs for Claude Code with each release, yet this one wasn't mentioned when introduced on March 26, and it was right around a huge user influx causing capacity issues.
This is what I've been wondering. I mean we all know they are compute constrained so I get that intuition to a point.
However, how would less compute result in degraded performance as opposed to more latency? Like is amount of compute something you can actually tune to control for latency? Wouldn't that imply they are actually serving requests in other models? If I run a very large model locally on a shit computer vs. something maxed out, it's the latency that differs.
The major providers like Azure along with OpenAI and Anthropic have the concept of "priority processing" which comes with service level guarantees on latency and token throughput. It's more expensive.
They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.
Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.
So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.
Thank you for the information but I'm not sure you understand my question. I'm not surprise to hear these details on how they control for latency.
Im asking on how lower compute could translate to worse model performance. That's the implicit claim being made when people are saying Anthropic nerfs a model (result is bad, not necessarily slow). The only way I can think that could happen is if they actually swapped Opus 5.5 out for a worse model that takes less resources to do inference.
Yes, that's what they are claiming; that the full(er) precision models get heavily quantized and/or distilled to lesser, "nerfed" models and requests get routed to them based on load, time since release, time to new model release, and reading chicken bones.
There is no proof, the frontier labs deny it, and certainly it's not happening on hosts like Azure, Amazon, and etc.. The claim is that it's "trivial" to detect observation, so they serve the normal model if you try to measure it..
Basically the conspricy theory is:
* OpenAI/Anthropic nerf their models regularly to reduce compute demand
I've seen big drops in Opus 5.5 (on high/x-high) & Fable. Both in terms of following instructions / workflows execution and overall code quality. Also the token burn-rate increased a lot, I've reached limits of 3 Ultra subscriptions during last week & had to spend 1.5k on API usage during the weekend just to be frustrated with the stupidity of the output.
For me I've noticed the pattern of decline happening on Tuesday and Wednesdays. I find myself having to have Opus redo many steps and making a much larger number of errors on those days. I am able to be more productive on Thursdays just as my weekly reset approaches on Friday at 1 pm local time.
I don't have measured data aside from myself noticing my frustration level with Claude jumping on these days frequently over the past several months.
Yes. That was the weekend my token usage blew up and used up my weekly allowance in a matter of hours. It was so stark.
After I figured out, like this chap on X, that it wasn't ME, I diversified my model usage and now a lot more work has shifted to Astra/Sol and I'm learning to make my workflows more resistant to the whims of each model.
I am using Opus 5.5 high almost exclusively. I definitely felt this and looked for nerfing watch sites. It felt like downgrading to the old version of deepseek. Things have improved a bit now.
The reverse happen with Deepseek. Getting smarter and faster especially if you dont mind the China API. Gemini Argon getting dumber compare to older 3.8 and significantly slower too.
The way Anthropic continues to get away with testing these nerfing "experiments" on customers without their consent depriving them of the model quality they've paid for is wrong.
The Windows Insider program has shown plenty of people will willingly sign up to be testers for a taste of the rawhide. Anthropic could do this in a transparent way but they continue to choose not too.
They can get away with it because they are so far ahead of OpenAI models for coding, it's not like you can switch to anything else if you don't like it. Astra is even worse than nerfed 5.5.
Pretty much the only alternative is loading up some credits and using Fable.
Astra when it hit the ground felt like “A Fable monent”: it was so good and concise and just was done with whatever task in a short time, where opus and fable sometimes can take forever. I think it lasted a week or two until it started to deteriorate to where we are now. Coincided nicely with the stopping of codex20 subscriptions etc.
Sol/Astra are sweet but they also have pretty much no discipline or executive function, which is where Opus 5.5 comes in, and I've not seen that capability in it nerfed too bad in my experience.
Every time I read something like this that doesn't match my experience I'm bewildered until I remember that providers play quality games and that everybody's workload and harness is different. For me GPT 6.1 Sol is tremendous value and excellent at low-level work like DSP and 3D game engines whereas Opus is more expensive and not as capable. I also don't get along with DeepSeek at all even though everybody on HN seems to think it's the second coming.
A neutral third party who runs a matrix of closed-source coding benchmarks across multiple harnesses would be helpful but that'd take a lot of time and money...
My experience is that 6.1 Sol is a little behind Opus 5.5 but it’s in spitting range. Their cost per task is generally higher, but if you’re on a quota plan the difference in cost doesn’t really matter until you’re paying API rates.
Both get the job done these days. I still hate Opus 5.5’s output, 6.1 Sol does an acceptable job of doing a second cleanup pass to it.
Yes this is the experiment I have been running for a while, that shows the constant comments on Claude models here have to be push from Claude team friends...The experience of output of OpenAI models is consistently better.
1)- Get a coding task and get the output of Opus and Astra.
2)- Open a new session and ask each other to review in detail the solution from the other model without mentioning where is it coming from.
3)- After the review, ask the original model to comment on it and state which solution is better and if the criticisms are valid...all in a new session.
4)- At the end Opus and others from Anthropic always end up agreeing the OpenAI model solution is better. My human review agrees.
After doing this for a few weeks...and consistently having their models rating higher the OpenAI solutions...I cancelled the Anthropic subscription...
Isn't this inherently task/domain/programming language/even prompting style specific? Why can't other people have genuinely different experience than you?
Yep, just one hour ago I again reached my weekly limit on claude code (3 days till reset btw), using Fable as Head-Orchestrator + Codex sub-agents via custom mcp.
I tried $20 of usage credits as a test once the limit was reached.
I shit you not, it lasted for less than 5 minutes.
Yeah it's a compete joke/scam. I bought $200 of credits once... gone in an hour. Never again. Part of me is thinking they decreased usage partly to force users to buy credits before the IPO... but at least in this case they ended up losing a user.
I get 5x more done (of equivalent quality) on my opencode go + DeepSeek flash sub for $40, than $200 anthropic sub.
Yeah it works great, maybe not quite as good as opus 5, but it's literally 20x-50x cheaper, and several times faster, an extra review/refinement loop per task brings the work up to opus level while still being much faster and cheaper overall. I have my own setup/tool (which I hope to open source soon) for planning/tasking/tracking which I found helpful to keep things on track an not forget/redo things - regardless of model - essentially tasks, dependencies/hierarchy, priorities, requirements, and task completion evidence tracked in the repo in structured plaintext/markdown files. DeepSeek was able to very easily understand and follow the workflow right off the bat.
Ok, thanks, I'm going to try to setup the pi harness and add on some needed extensions for my workflow. I'd appreaciate any advice in regards to any addons you would recommend if you have them.
Eh sometimes it cuts the other way, like when Opus 5.5 was being shadow-routed on Opus 5. I am on a team plan and seem to not get any of these A/B tests.
They should let you use your subscription with your own harness. Until they do, you're just borrowing someone else's workflow, which is liable and likely to change whenever the fellas over at Anthropic get a wild hair and decide to introduce a new experiment.
I agree, I also noticed visible drop in quality, a week before I was able to ship two apps and when I was working on the third (which is almost as much big as the first) it started acting weird, I even started a new session thinking it was because of hallucination... It did give me the output but burnt through my tokens earlier than I expected. It was even missing and getting confused between branches... All in all doing more mistakes... One point I would like to bring to your attention is, it has started using background agents a lot lately (maybe because I prompted it to use Sonnet models for scripting and Opus will supervise, but still it was better in the earliest week.)
We use Azure Foundry hosted Opus and it seems as capable as ever.
I run a small local GPU cluster for Qwen 3.8 Flash and while it's "free" (you should check electricity prices in EU...) unfortunately it doesn't even compare to Opus 5.5 or Astra in capabilities. It's like GPT 6 vs GPT ~4, but imagine GPT 4 is also 100x slower and sounds like an airplane taking off.
In the end it's totally possible that I would pay more for electricity running Qwen for 24 hours churning on some complex problem vs. Astra one-shotting it in 15 minutes.
I hope we get there, but it's not there yet, and I'm sure never will be with consumer GPUs. Maybe if you spend $20k+ on a local AI rig. There's just no market force to give access to AI to everyone at fair price.
Sadly this thing stopped running a few weeks ago, the propduct half life is truly low these days otherwise wed have some data to rely on: https://marginlab.ai/trackers/claude-code/
This is not real model's reasoning chain, they might have just replaced or tuned the model that summarizes it. If you want to track actual degradation you need to run evals. Trackers like this do exist, e.g. https://aistupidlevel.info/
Anecdotal, but I tend to get by with medium thinking on Opus and I noticed that as of sometime last week it really did feel like it had become noticeably slower to think about things.
I wouldn't say I've especially noticed its effectiveness having been reduced, it's just slower.
We need a consistent way to tell level of service for LLM services. Without they can play with us and steal money by providing different/quantized models than we paid for. I believe this may be a good next AI company if they will be reliable.
I've noticed I've been running into more safety guard rails the last few days. I've been using fable 5.1 for scientific lit search (e.g., on topic X, give me a table with links to the top 20 papers, and then major criticisms and rebuttals with direct links to the papers, etc.) to just save me time in getting acquainted with a new sub-topic. The last couple of days it's not just been dropping me from 5.1 to opus 5.5, but all the way to opus 5, stating that it's for safety reasons.
I have noticed a mass lobotomy of Anthropic models recently. The frustrating part is that the target keeps changing and however you work with it today, it can fail tomorrow.
Hm, I’ve seen a degradation of 5.5 yesterday. I’ve had a session running for around 50 days and it just started to generate garbage and ignored all rules. A fresh one feels the same as always
How can anyone build on this quicksand where you never know whether the same model will still be available tomorrow? The whole industry has become a casino.
What I'm surprised by is that no-one has mentioned that FAANG companies run experiments (often multiple overlapping ones) on 99% of the traffic. You're almost never getting "vanilla". And Anthropic is full of talent from those places.
If you look at any sort of tracker, there's no telling whether you're in the same experiment group as the tracker (most likely not).
this is the reason i cancelled claude after 4.6, non-determinism aside, anthropic just doesn't seem care about maintaining sane baseline for their products
This could explain needing to blow through my limits last week. So much of it’s outputs on xhigh needed serious revision, and took much longer than usual to meet specs.
I’m only prototyping so most of the code will be discarded, but even in this circumstances you can’t allow too much sprawl and decoherence within a single project. It makes prototyping cost so much more over time. It was pretty frustrating.
Sometimes I feel like the model is not necessarily needed, but the harness might be. Asking questions/permissions more often, delegating less, creating fewer sub agents. Just overall making workflows slower/harder to review. The main model might be just as smart/fast, bur it’s like the agentic side that starts crapping out
If Anthropic indeed nerfed Opus 5.5 on November 3, it could be because they wanted to be prepared for inference cost going up due to the data center moratoriums and the election results. I really hope this is false, but I don't see how inference cost can go down in the near future.
1) it was certain for a long time before the election that the unwholesome party is going to get slaughtered
2) the wholesome party is not in power yet
3) even when they are, the orange man will veto everything they might attempt to do
4) even when the orange man is replaced with someone wholesome, and that wholesome person enacts a bunch of wholesome legislations, data centers will simply move literally anywhere else on Earth.
Regarding 4, I don't think anyone in the US, regardless of how they voted, would want a bunch of SOTA GPUs being in any other country than the US. It's either they are all here in the US, or they are not anywhere. As a result, if data centers cannot be built in the US, the GPUs will not be produced only to be exported elsewhere.
>In Ohio alone, at least 14 municipalities passed bans or restrictions on datacenter development, including in Sunbury, where Amazon has proposed a $2bn datacenter project. In Maine, eight municipalities passed temporary datacenter moratoriums. In Emporia, Kansas, where a 1,000-acre gigawatt-scale datacenter was recently proposed, a ban on high-impact datacenters was approved by voters.
and now they will be built elsewhere.
I think there's a rather obvious precedent for that inevitable outcome.
I mean, sure, they'll be built somewhere else (and I think the approach of "stopping" rather than "protecting residents from price changes" is terrible), but buying land, permitting, etc take years and tons of money. It's going to slow down the industry a bit.
It has been nerfed for a long time now. Anyone who tried original Fable 5, knows how good that was, and all subsequent models, including current Fable and Opus have definitely been nerfed. It's like they add a mandatory 5% error added to all code, etc...
It's very frustrating, even at max effort. It used to do things correctly on its own, now I need to keep reminding it to do obvious things like "Make sure you are not dropping steps as you rewrite the docs". It became unreliable, like how Opus 5 was.
You have to keep the model unchanged from when you released it. Changes have to be mandated to roll over to a new version. Otherwise all of this is useless, you have to atleast be “deterministic” with the model capabilities.
There's this ongoing theory that Anthropic makes it's model dumber on purpose to make the difference noticeable with new model release. It has been documented by someone, you could see performance being dropped
Its coming a time that these commercial offerings will have to provide independent daily benchmarking to ensure their numbers align every day, with the risk of being called out for defrauding customers.
Personally, I feel that Opus 5.5 is starting to behave like the old Opus 5. Lately, the model has been using a lot more of the compact, hard-to-read jargon that is typical in Opus 5
If this is real, it is straight up violation of implicit contract. When you pay for a service you expect stable and reliable performance. Intentional degradation of service is basically fraud.
I have a small eval set that I rerun every week with my own tasks. That's the only way I've found so far to distinguish between model changes and having a bad day.
I feel strongly that GPT-6 Luna has been nerfed. I was super-impressed by it in the beginning, now it seems to think overly much and get absolutely nowhere with even very simple tasks.
It's hard to tell, but it felt the same to me. Either way, Fable 5.2 just came out so abnormalities might have caused by infra-side changes related to that.
LLMs are such a great business, they are non-deterministic, hence you can do whatever you want, because customers can't do anything about it:
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
You can say the same thing about nearly any business. Lyft and Uber, for instance. You can lower quality, raise price, whatever. This is why open markets matter so much - when you have competition, businesses can't materially worsen their customers' outcomes for long or those customers switch.
How do you even lower the quality - serve half broken car for a luxury order? No way person who orders luxury car accepts it.
Or do you increase wait times? - sure, order gets cancelled, you won't get a payment for that.
There are of course ways, but you still can do something, in case of LLMs, you blame yourself for poor prompting and retry, which again makes money to LLM providers
- You can lower the quality standard of which car counts as which class. There are many classes of vehicle on both platforms.
- You can delete or not consider low ratings from certain users. Both companies do this already.
- You can increase wait times, of course. It doesn't at all follow that the order gets cancelled. It depends on the market and what each company believes the other is doing.
Uber/Lyft are infamous for using their control over both ends of the transaction to milk both ends for money. They have heavy use of manipulative dark patterns. They know the entire market on both ends so they can manipulate the spread to optimize profit for themselves... and they can hide much of it simply by not telling users.
Eagerly awaiting Artificial Analysis to redo their Anthropic benchmarks and kick it down 15-20 index points, in the interests of accuracy. Those guys love accuracy. They moved Heaven and Earth to kick Kimi off the #2 spot, out of their deep, abiding passion for having the most accurate benchmarks in the industry.
I have no doubt they will ensure their users get an accurate picture of Opus 5.5 within hours.
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's language from the EU "dual quality" food scandal, which saw food brands (Coca-Cola was one) adulterate their recipes to use cheaper ingredients in select markets—though the external packaging was kept the same.
If I were a cynic I’d also say that it wouldn’t be particularly difficult for them to detect when a benchmark/eval context is active and to route those requests to larger models or higher quants.
This'll likely be the reality moving forward - they allocate more compute at launch to get good test scores, reviews, and news coverge, they get people excited and used to it, then they cut costs.
I'm reminded of the video game industry where they would launch a game without any in-game purchases at launch, get the reviews to make note of it and give it a higher rating because of it... then add in-game purchases anyway. I think that stopped again though.
I look at these threads as the other side of the "LLMs are super-intelligence" movement. If my superhuman champion has disappointed me in some way, it must be nefarious interference. The unpredictability of LLMs is immanent in the technology, but the movement elides this for its own purposes.
It’s A/B experiments at scale to see how many people put up with endless quantisation, a defence mechanism so they can say “hey all these wildly unjustifiable salaries for developers and management are totally unwild and justifiable, why look at how much we’ve slashed our inference costs”
Sounds like analysis fully driven by a model, which can be fine, if you know what you are getting into and can verify the output. Just last week had someone ask Claude Opus 5.5 on High about watermarking after I informed him and he'd rather believe a models output over me, do a basic Google search or spend the effort reading Anthropics communications on the matter.
The model told him with no uncertainty that watermarking text was not implemented in any Anthropic models, whereupon he showed me and I then asked him to just ask the model to check again, ideally online. These models do still "hallucinate", they still have major gaps in their output not easily caught if you don't know the topic and interestingly, anything connected to LLMs, model providers, etc. tends to be where they do worst.
The annoying thing is, they still sound immensely authoritative across most any topic, but especially LLM stuff, which is such an easy way to get tricked. I have been for sure in the past and will be in the future, it's why I recommend having some incredibly niche hobby (like sewing with less known materials, obscure motorcycles, etc.) to check how quickly the model makes a mistake which no one in the know would make. Just to keep yourself disillusioned and inoculated.
In this case, looks like the LLM did not consider that these are (and have been for months now) just summaries and the actual reasoning can not be analyzed for encryption reasons.
If in doubt, please do some research independent of the model output. LLMs not having any internal "understanding" that'd give them anything akin to an internalized "feel" for what they "know" accurately enough is one of their major flaws.
Funnily enough, once did the opposite, made a manual analysis of a Github outage, then had the model just write an artifact to quickly share. Was lazy, could have just put something on Pages, but taking summary changes and extrapolating that to a "nerf" is less ideal.
On the regression front, I run my own, private evals and have only once encountered consistently reproducible, over run-to-run deviation, degraded performance, which was purely present when using Claude Opus 4.8 in April after some system prompt changes in Claude Code specifically which were addressed soon thereafter. These were not present with OpenCode via API and any other time the public on Reddit and co. reported regressions, I could not reproduce it, even when using Claude Code as the harness and a subscription.
What I have encountered and been able to reliably reproduce is that certain labs do very clear teach to the public benchmarks and underperform them commonly, whereas others do not. And that certain, especially cheaper and higher throughput, neoclouds very clearly take some quality affecting, tool call misformatting shortcuts.
For what it's worth, currently measuring, but haven't seen anything so far with Opus 5.5, especially nothing connected to reasoning that'd make the assessments some of the commenters draw from this apparent.
My paranoia tells me they A/B test users which means consistent benchmarks go through the high performance pipeline while regular requests fall back to the quantized / experimental / low cost version. I can see the main reason being the oldest one in the book, which is making more money.
It could also be a regional effect where systems are under higher load and hence dynamically routing to / provisioning lower precision instances; or some other factor where one is affected whereas someone else isn't.
Isn't this just the tried-and-true planned obsolescence playbook? The frontier labs need to capture as much value as they can from the next "hit", and that includes eliminating competition where they can. Including when it's their own model. I have zero evidence for this, but it seems patently obvious: Nerf Claude to drive usage to their next model, where they compound value and (theoretically) can serve more efficiently with each generation.
Anecdotally, it's currently feeling nerfed. The model keeps restating my intent as if it had thought of it. Output accuracy is slightly down, but the increased friction in communication is having an impact on my worfklow. Very annoying. Also, it flat out ignored system claude.md several times over the weekend. These observations hold between my personal pro and works enterprise.
I use Sonnet 5.5 daily and Opus rarely. My usage is also a little involved as in I do all the design and strategy and get Sonnet to just write the code
For that, Sonnet has been consistent
I use Opus or Fable when it's a dumpster fire situation where we have no time to think even
These kinds of threads always make me feel like I'm watching some satire. Pure consumerism, like the people that used to fight in line on Black Friday. Maybe Anthropic is nerfing the models, why are you so addicted to them that you care?
Anecdotes and conspiracies and mass hysteria abound imo. And probably a bit of astroturfing from competitors. Models are not deterministic, nor are your projects. I work on code constantly for years - and across every anthropic and openai model people claim the same thing - yet I've never experienced this degredation, not even one hint of it. I would be acutely aware of it.
Variable token usage? Sure. To be expected in non-deterministic systems like these.
Distinct increase in errors and issues? Never. Not within a single model's lifespan have I ever experienced this.
And this whole notion of the A/B testing of performance/usage is completely ridiculous. There is zero evidence it has anything to do with performance. There are a trillion things to A/B test besides things that hamstring groups of users, so this is just being used to confirm the bias of people already caught up in these "theories."
Perhaps most importantly, human bias is not deterministic either. Everything seems consistent with people getting excited by the hype (or just the novelty) when a new model is released, and then coming back to reality a little while later.
If they're messing with models like this, surely they'd at least occasionally make a tweak that improves intelligence rather than degrading it. But I never see anyone say Model 4.2 got smarter a week ago or whatever.
I also noticed a change. Normally it said, "Found XYZ. Now I do ABC". Now it says "Found XYZ. Now I will do ABC." I liked the robotic "Now I do ABC" better.
Say they did nerf it, why would they tell you? Why would they tell anyone?
Or maybe they didn't nerf it, and it's the same as yesterday, but you had a spate of bad sessions due to the whims of probability and statistics - why would they bother telling you that nothing has changed and the problem is you're unlucky? Do you think they want to remind you your experience of their product might be crap and no one can do anything about it? Do you think you'll take kindly to that? Do you think in the large, that a large number of people who aren't you would largely take kindly to receiving such news? Do they want that support burden? (They don't.)
Are we understanding statistics yet? You're only one person in a sea of potentially affected or non-affected people who are each potentially affected to different degrees or not at all, and it's a probabilistic machine they're selling which is used enough that, due solely to probability, some users will roll snake eyes for days.
Say that actually all this was wrong and actually, they ARE A/B testing and that caused SOME people to get unintentionally nerfed as a side-effect of one test (how many tests are underway at once by the way?). Why would they ever admit to that? "We noticed you got non-consentually subjected to our product-improvement bullshit. Nothing you can do! It's in the EULA!"
Besides which how would they even know who was affected? If they did, would they do a targeted apology, or apologise to everyone just in case?
(There's precedent here: when the thinking level default was lowered in Claude Code, there WAS a noticeable and broad effect, and there was a (admittedly quiet) apology and a fix issued. There was also a fix issued when the CC prompt got insanely wordy - the "terse" tone of voice mode, a new flag you had to update to get, and set manually.)
Let's bring in some game theory.
Let's put ourselves in an alternative hypothetical universe where everyone is going for the "trust angle" and thus publishes numbers proving non-nerfing. Initially every provider spends time, money, and energy - but the industry is so competitive so every lab needs every bum in every seat working on the product, so eventually SOMEONE stops and then there's no incentive for anyone else to keep doing it. Know what happens? The whole thing collapses. Being public with non-nerfing claims precludes A/B testing, putting whoever can't A/B test at a disadvantage. And also, when the numbers get withdrawn after people expected to see them it's a super bad look. So why would anyone bother in the first place? We're back to our own universe.
Silence is signal.
Do yourself a favor and ignore rigor-free superstition and anecdotally driven frustration and kvetching from folks with NO skin in the game.
ESPECIALLY anyone who misunderstands probability while at the helm of a probabilistic machine.
The more you listen to people tell you stories based on one single experience without any basis in statistics, the more you'll believe things rooted in opinion over things rooted in truth. That way lies madness. Steer clear!
I thinks more likely ya'll are experiencing a phenomenon that has been discussed on here before.
At the model release people become high/manic/addicted to the newest model think they now have some new superpower and then that "high" wears off and you think the model isnt as impressive.
The truth is llms have a slot machine-esque UX to them, and the labs designed it this way.
Dont worry you'll get your new model with some mystical name any day now and a month or two afterwards you'll be posting about how it no longer feels as good to use.
Each new model version hits your dopamine receptors very hard when it first comes out, but in the nature of things it stops hitting after a while and you need novel models to fire them off again. That's why people think the models are getting nerfed after a couple of weeks; it's actually their endocrine system getting nerfed.
There were no noticeable complaints about Opus 5.5 before the end of last week. Large volumes of people are now reporting the same thing at the same time. If your theory is true then multitudes of endocrine systems separated by vast distances and vastly different initial Anthropic exposure times are perfectly in sync. Resistance is futile, you will be assimilated.
One of the best scenes in The Wire was when Avon explained to D'Angelo that the new shipment of heroin was just the old shipment of heroin cut with different baby powder.
Is there a practical reason this link uses twitter.com rather than x.com, or is it a nod to the old name? If it’s the latter, I don’t really see the point.
IPO is coming so they will have to maximize profits going forward which will mean constraining the compute they give everyone to ensure the subscriptions make money.
By selling subscription tokens for less than they cost to generate, of course.
What you said doesn't yield any information whatsoever about whether or not subscriptions are profitable, all that says is that subscriptions are less profitable than API.
Things are able to be less profitable than other things while still being profitable.
If something like the API is very very profitable, it's even possible for profitable things to be much less profitable!
Tokens are tokens, whether they're used on a subscription or via API.
When you purchase a subscription, you get a certain amount of usage. You're not told what that usage level is.
You do know how many tokens you use.
So when you calculate how many tokens you've used on a fixed-rate subscription, and use API costs to estimate how much it would have cost you without a subscription, you get what I referred to as the API cost of the token, or what people are actually paying for which would seem to give an indication as to the "market price". That means the price the market will pay for it.
I don't have any evidence/study to substantiate this but I have the feeling there's a _massive_ amount of fiddling with the quant/functionality of the models at different stages of the release cycle and account freshness.
Every time a new model drops it feels wildly powerful, then regresses to some sort of generic SOTA level we've been used to over the last 6 months.
I remember my first coding prompt with, I think it was sonnet 4.6, on a fresh account and it was shocking how much it successfully did off a single prompt - I haven't found that with any other model at any other time and I suspect it was because they put me at the top of the priority list to get a "wow" reaction.
Regression after a release is pretty well-agreed upon, but I'm wondering if they secretly have a huge amount in the tank they aren't releasing. Either as a card to play in the future, or because they can't keep up with demand. I've heard anecdotes of people working in these companies saying that the public models aren't even close to their actual capabilities... Anyone have any opinion on this?
So, IMO, they have much better capability internally because they have 'unlimited' usage. I also suspect a lot of 'Fable' is just a shit ton of nested subagents doing necessary backend work with a very solid fine tuned LLM conversing with the user.
I worked recently for a German megacorp, for whatever reason they gave us unlimited access to a bunch of models, up to sonnet 4.5/Gpt o1 etc, so not the best or latest models. And no agentic tools.
So I built a custom Claude Code clone because they exposed the api from their custom 'LLM Chat' clone in the client and accepted my user credentials for access.
Essentially a frankenstein AI harness using various models for their strengths, including using the OpenAI best model for memory context consolidation and compaction + Sonnet as orchestrator, etc. I was able to spawn 20+ agents to do my work and basically automate my entire job at that corp. I think my internal bill was something like 5k euros per month and I was being careful to not abuse it...
It was better than the best you could get from claude at the time (about a year ago now I think), in terms of task performance completion.
So imagine what someone with essentially infinite spend can do with endless fable agents. AKA US government and Anthropic internal.
> I was able to spawn 20+ agents to do my work and basically automate my entire job at that corp
Not a great plan
Not if you want to stay at the job. I quit to work full time on harness development.
whats your harness called?
I'm not releasing a harness as standalone, rather, I am creating custom harnesses internally for 2 different SAAS projects (I'm hoping to release both by the end of the year). One is related to job app/job search, the other is 3d model and 3d cad custom generation from images/text description for jewlery.
Primarily I am doing this to not have to rely on the more expensive SOTA providers and have been tweaking the harness prompts and toolsets as needed + composing agents as tools (like a Jewlery Reviewer/Validator agent getting called on a worker agents work, Engraver agent etc.) and using DeepSeek flash for everything I can.
DeepSeek is so far more than capable for everything I need it to do for both projects.
Also for fun on the side I'm making a buy/sale shopkeeper fps that is using internal harnesses to drive character ai + conversations. Something kind of like 'Probably Stolen' but you actually have to use voice/type everything to a dynamic npc.
I guess doing pantomimes for 'job security' has become so accepted and mainstream that it doesn't rub people the wrong way.
I'd rather just do the best job I possibly can, with the tools available.
I need money, but I also can’t live with myself if I know I’m doing something the wrong way.
The thing that bothers me is the false advertising. The models are announced with specific capabilities and benchmark scores. Then they silently nerf the models after release. Without a public acknowledgement that a production model has been nerfed and what the new benchmarks are, people are assuming the model has the original capabilities. People are choosing to pay for models based on their reported capabilities. Customers trust these models to deliver the reported capability. It feels like false advertising to me when they say one thing and deliver something less silently hoping no one finds out.
If I order a 16oz beer and get a 12oz pour, I'm going to call it out.
I wonder if the questions about the quality in claude code have something to do with it. Like as long as you say it's great, they can take little steps dialing the effort back.
> If I order a 16oz beer and get a 12oz pour, I'm going to call it out.
And then Sam Altman is gonna sue your ass for owning a measuring cup.
[flagged]
It's kind of crazy they can do this without informing customers. Essentially we like many other knowledge working companies are slowly outsourcing intelligence to a 3rd party that can, at will it seems, alter that intelligence without informing us. Just means we have to invest more in harnesses, guardrails and keep active in open model space.
Google isn't informing customers everytime it updates its algorithm
1. Two things can suck independently at the same time. One thing sucking is not a justification for another thing sucking.
2. Google doesn't pretend that its algorithm has certain capabilities while serving the user a nerfed version
I think (2) is the important one. Anthropic provides benchmarks with their model releases. If they are making changes that make models perform worse on advertised benchmarks, how can this be legal?
It's a good point. I mean I have no idea if they are actually breaking the law, would assume their tc's would dissolve them from any legal jeopardy but it's still not right.
Is this where independent benchmark companies like artificial analysis could help ?
In fact, people accepting that the new thing can suck is why newer things suck more. That's how the frog boils.
Google gives a lot of information to its customers, the advertisers, when it changes things that impact them a lot. Not enough for all of them to be happy, but it does private and public communications on big changes.
caveat emptor
But like, you wouldn't say that about other services where we trust the provider out of necessity, like engineering or pharma.
Seems to me that imposing reputational consequences on supply shenanigans is one of the key ways that an emptor may cavere.
> they secretly have a huge amount in the tank they aren't releasing
I think it's more how much immediate profit they are willing to forgo today per account age, subscription tier, and usage type. Burning more compute means they make less or nothing (or have to subsidize) vs dialing up increasing amounts of the ever-more aggressive 'optimizations' (aka nerfs) required to get them closer to viable profitability after their IPOs.
They have dozens of 'knobs' which are completely opaque to us yet each indirectly trades worse results for lower cost. On the other side, they MUST show a clear path to HUGE top-line revenue growth and bottom line profit growth from where they are today by min/maxing new account retention AND renewals-after-price-increases and continual nerfs.
We're not using a somewhat variable, load balanced cloud service, we're in a carefully engineered Pavlovian Monte Carlo casino. There are at least 1,000 different Claudes, Sols and Astras at any given moment. Every time we start a session we're spinning a behavioral roulette wheel. In effect, we're the crabs in a pot of water that's being pushed ever-closer to boiling while they're testing to ensure not too many of us jump out per degree increase (they're fine if the less profitable ones jump out). This will inevitably worsen since they are still a long way from justifying their valuations.
This regression in Opus 5.5 started for me on Monday (rolled out Sunday night?). It's very obvious.
I assumed before the IPO the goal would be to prove that the models are fundamental, capable of building deep complex tasks. But more: prove that they have engineers believing in the value of the tool as offered by ANT(could also frame it as "engineers believe the cost is worth it").
If we are having this discussion about nerfs and regressions before the company even IPO'ed, it feels like they are missing the boat on wooing engineers for the long game.
If the sentiment is not "i can't wait for every model release, they just keep building these tools that are so incredible" but instead "I can't believe they nerfed it, I can't wait until something more dependable comes along", we won't see the jump ship right now. But it will happen.
> If we are having this discussion about nerfs and regressions before the company even IPO'ed, it feels like they are missing the boat on wooing engineers for the long game.
The super-linear scaling rate of capability-per-dollar started tapering back toward linear sooner than they hoped. They can no longer afford to subsidize wooing engineers, nor allow today's primary revenue customers to be less than 'Wall Street Expectation' profitable.
It would be nice if Anthropic would be more transparent about these things.
I'll gladly sign up to be a "Experimental Settings" beta tester (with knobs of how extreme I want to go) for a slight discount (or just bill me the same, just gift me a reset every week? idk whatever).
Doing A/B testing in prod on a frontier model that could really mess with the users in really bad ways is a bad idea.
Why would they ever do this instead of just rolling things out like they are now? What is the incentive to give you anything when they can move faster and just get the same result without it?
Because sometimes when they do these things some users are affected for weeks and leave for competitors, I guess they don't care as long as they don't have to be beholden to stock investors, but eventually that too will matter.
Don't you think that this is a calculated trade-off? You can roll out the A/B testing to 2%, or 10%, or 50%. There are statistics on all this. Individual user feedback is irrelevant.
> Individual user feedback is irrelevant.
Like Windows 8?
Windows 8 is pre-big data and massive scale data collection of every aspect of the human.
They had do rely on this kind of feedback. Now this is all automated, engagement goes down -- something should be fixed
There's direct evidence - OpenAI employees on twitter (tibo) flatly say they are constantly adjusting models after release, and even specifically state "improvements" to efficiency etc for models post-release. They presumably release to maximize benchmarks at the start, then make them "more efficient", and if you want to give them the benefit of the doubt, then presumably more efficient with negligible loss in benchmark scores.
For a while, Codex was accidentally truncating its thinking at multiples of 516 tokens: https://github.com/openai/codex/issues/29353
Because the codex harness and the intelligence tests are separate tools, you can end up in situations like this where A/B tests inadvertently break the tool customers actually use, while the tests say everything is awesome.
I bet some engineer got a bonus for “halving token usage” through “efficiency improvements” — announced around the same time — and not realising that they’ve accidentally lobotomised their model in the process.
“Oops.”
Also, it becomes really difficult to unwind errors like this because the “savings” get burned into profit and loss reports, S-1 filings, and so on.
Can you imagine that boardroom conversation!?
I feel like I am taking crazy pills when reading the other responses, but come on, this seems like normal human psychology when being exposed to a new thing over and over. It feels new, it solves everything, but then over time you get bored with it. Attempting to proxy it with "account freshness" just masks the problem. It's the reason why people tried to quantify it with stuff like https://github.com/ninjahawk/livenerf because the pattern you are referring to is a known bias.
https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner... ---> Interesting reddit thread going over what appears to be evidence of this kind of behavior from Anthropic.
You're telling me HN users have biases just like everyone else and engage in conspiratorial thinking or reverse engineering "rational" justifications for whatever they're feeling in the moment? no way ... /s
I noticed an enormous drop in quality around this time.
Opus 5.5 was significantly crushing every task I gave it for quite some time.
Suddenly, it starts making the type of mistakes that even Sol 5.6 would not make. It became noticeably worse than Astra.
I've grown so completely tired of the labs changing the quality of the model out from under our feet that I CANNOT wait until there's viable alternatives at the frontier level.
I will gladly forgo the best available power for consistency - and I see that coming soon (hopefully with Qwen 4 or shortly after).
DeepSeek v4.1 Flash is not even remotely in the same league as an un-nerfed Astra or Opus 5.5 - despite what microbenchmarks may claim.
It's a great model, especially for the cost - but there's a huge difference.
Same. I'm doing something security related right now and I swear it was smarter than this before. Right now it keeps pushing me to just allow all connections, or leave a password in plain text, or other insane suggestions. Maybe placebo, maybe bad timing though, but it feels ridiculous.
Edit: Just to add some more ridiculousness:
> The last squash stalled because it opened an editor to merge descriptions. I've stopped it. Checking what landed:
I was using jj with Claude for a while now, suddenly it forgot how to squash? It also started doing that stupid thing where it edits files through python scripts so I don't see the diff during manual mode. This is like 3.5 levels of stupid.
If this is in Claude Code, check whether your session got switched to Opus 4.8. After a cyber safeguards refusal it continues on claude-opus-4-8 with a one-line notice. I saw this on ordinary auth-token fixes, in 5 of 60 runs last week.
That is a setting. The first time you get refused it offers to stop or switch to 4.8 and if you choose 4.8 it automatically downgrades silently whenever it happens again. I just shut it down and switch to Codex when this happens.
I would also agree that 5.5 feels dumber the past week. More weird logical problems. it's not as dumb as 5.1 or 4.8 but it feels like it regressed.
I'd started looking at https://marginlab.ai/trackers/claude-code-historical-perform... this week because doing the same tasks a fortnight ago versus this week and even today I was having to argue with or question everything from Opus 5.5 on medium.
The link hasn't been updated since September but anecdotally I had Claude today when asked to write something create a 'to-do' and instructions for me to write a code block.
Another example is suggesting tailscale serve to proxy from one address:port to a different port (so still address:port) when setting up a reverse proxy.
I can't help but think about the parallels with some types of drug use, where user's report chasing their first high but never achieving it.
Whether true or not, this kind of sentiment will eventually destroy their business model with users preferring local models instead. When they are good enough.
Fable 5.2 just dropped!
https://news.ycombinator.com/item?id=50429947
> Either as a card to play in the future, or because they can't keep up with demand.
I suspect it's rather a simple matter of cost.
> I don't have any evidence/study to substantiate this but I have the feeling there's a _massive_ amount of fiddling with the quant/functionality of the models at different stages of the release cycle and account freshness.
Oh good Lord, they stir the pots constantly. Anthropic is hella naughty about this. Someone else on HN said it well, Anthropic has too much of a "researcher" mindset. You cant experiment on human populations tool use the same way you run AI/ML experiments.
I want a stable tool that works consistently, not a hammer that changes mid-swing (credit Mario Zechner for this analogy).
> You cant experiment on human populations tool use the same way you run AI/ML experiments.
Google and Facebook have proven otherwise. Unless you are very, very lucky you are always part of multiple running experiments on their products, and nobody really cares.
If it's mostly indiscernible from baseline that's perfectly fine and allows for incremental improvement. Think infinitesimal changes in light of multiple dimensions. When you have separate development and stable versions there's always the risk of packing too many large changes and no one actually wanting to upgrade (given a choice) because the product is already good enough. Pro and con for both approaches but IMO, you need mostly small changes and occasional big steps when small ones can't get you there.
They probably keep it at "full power" when released because that's when most people will be running benchmarks, then cripple it to save money.
In a just world this would be considered consumer fraud, or defrauding investors. Things that if you read the law you would imagine that it came with tens or hundreds of millions of dollars in fines and/or jail time for everyone up and down the chain.
If you're selling model capabilities for money, and you change the mobile capabilities, not changing the model designator (name and/or version number) is fraud, full stop.
A VW engineer got jail time for Dieselgate, right?
Hopefully that was a manager and not an engineer otherwise I'm fearful of doing any engineering in ${conviction location} (Germany?)
Hopefully it was every involved from top to bottom. Crime is crime.
And yes, if you're doing engineering that violates the law, you should be fearful of doing. That's kind of the point of punishing criminals.
> A Volkswagen engineer was sentenced today by U.S. District Judge Sean F. Cox of the Eastern District of Michigan to 40 months in federal prison, and two years of supervised release, for his role in a nearly 10-year conspiracy to defraud U.S. regulators and Volkswagen customers
https://www.justice.gov/archives/opa/pr/volkswagen-engineer-...
Mindblowingly, the CEO and 4 execs.
Unlike here where the "phantom bank accounts" fraud was successfully pinned on tellers and branch managers who were all "independently" just trying to meet management goals...
DeepSeek flash doesnt have this problem. Im routinely astounded how much i was being gypoed
It's more likely it follows the compute demands of the development cycle.
> Every time a new model drops it feels wildly powerful, then regresses to some sort of generic SOTA level we've been used to over the last 6 months.
Is this 2026's "my macbook feels 'snappy' when I update it", though? (That is to say, not really much in evidence)
House of El covered this in her latest video incidentally.
I read this so often - why is there no research / whistleblowing around this? The incentives to do so will be high after the IPO at the latest
I agree, it seems like they try to find optimization twoards a baseline short and long term, and maybe one for transition time for a new model.
I had assumed this was just me. I keep seeing the models do things nearly magical upfront, and then regress is subsequent weeks.
Last night I was working on datalog, but Fable was acting stupid and its responses were LLM babel. After every response, my reaction was "What?". I stopped using it. This morning I wake up to this in my Claude Code emacs session: "Why are you mixing up the datalog stuff in here? you're working on emacs." Claude Code replied, "I did the work the datalog session sent here instead of passing it to you."
Volkswagens new diesel powered LLM has great environmental statistics.
I'd venture to say they are about to drop Fable 5.5 or perhaps the next Opus version. Over the weekend, their current models were objectively much slower than usual and subjectively not very bright.
I don't know who they think they're fooling.
Edit: Fable 5.2, whatever. There you go.
> Every time a new model drops it feels wildly powerful, then regresses to some sort of generic SOTA level we've been used to over the last 6 months.
This isn't new to the current LLM bubble. Grammarly has been like this for 10 years.
For sure. I don't have any proof either but im 80% sure openai is gaslighting me with these model changes post-release
The local Qwen 3.8 27B might not be anywhere near as smart as Claude's models but I never wake up one morning and wonder if I am part of some test model to see if I will accept something somewhat less capable or not trained as well in the tools I am using day to day. Its also not suddenly going to start refusing to test the security of my system or scanning the code for vulnerabilities or continuing to back engineer that old binary the company lost its code for years ago.
It is something I also appreciate with some of the other labs like deepseek who publish the checkpoint they are running on and still sell you the old one for a period of time as well.
Its obnoxiously anticonsumer to be constantly chopping and changing what a model represents on customers who are paying them an awful lot more to be on the frontier models.
A slightly worse but consistent and reliable model is 100x better than a model that sometimes very good, then suddenly and secretly gets lobotomized.
While I agree with the general sentiment for a slightly worst model locally you need at minimum like 1 tb of ram.
Qwen 3.8 is useful and honestly impressive but it's far from slightly worse.
For slightly worse you need something like Kimi K3.1 or glm 5.3
isn't like 512GB enough? Pretty sure non kimi 3 class openweights run on 512
As much as I would like it to be true, I trust frontier models more even lobotomized. A good lawyer is better even after a beer
You don’t need to go local for that though do you? You could use a last gen open model hosted by a cloud provider that pins the version?
That defeats the whole purpose. You have no way of knowing that the version that the provider claims to be serving is the version that they are actually serving
You can always rent a box and serve an open-weight model to yourself remotely. That's probably the only economic solution that halfway makes sense, given hardware prices these days.
Have you looked into that claim? I dove into pricing for VPS instances and dedicated machines are like a magnitude more expensive.
Outright ownership is going to cost the least since you are responsible for amortizing the maintenance and replacement costs.
Providers also markup their rate because they assume customers are running paid services and can afford the markup.
Anthropic released a paper on zero trust verification of inference running on the same model.
You never know if youre really getting the offered quant. Only with local can you be sure.
Everybody says they pin. Meanwhile, if they can swap in a lower quant and serve more throughput and make more pennies, they do. It's easy to say, "well, KLD is at 5% so come on that's pretty much the same". And then that's the new norm for a bit, and then comes more traffic and they say, well, shit, let's do that again, it's only another 5%...
Majority of people who complain about nerfed models would do the same about local models.
It's the good old: leave a person on uninhabited island with 10 identical photos and after a month they would have found dozens of differences between them.
I’ve been coding for 20 years and like most don’t code anymore. I ship so much code it would take me weeks and months to review.
Does anyone fear that we accelerate so fast and depend on those super smart models, fire humans, that if at any point for any political or economic reasons the IA costs 100 or 1000 more or is shutdown … what will we do about our projects and maintenance of them ? Will we still have have enough people and time to treat urgent bugs and dig to understand everything and fix ?
I feel it and it makes me anxious some days that, if there is an important bug and I don’t have tokens anymore I’m doomed or I’ll have to pay expensive api price. I think about Europeans, if not any reason politics play and cut access to US models that we depend on for work, we’ll end up with subpar Mistral ones and that could be very hard for companies no ?
The frontier models are owned by large corporations that still have some accountability, and the engineers there probably see themselves generally as white-hat.
The question is, what happens when frontier models have nefarious ownership?
Like the recent OpenAI "We found 900 linux kernel CVEs", I think we could assume a nefarious frontier model owner would just start exploiting the bugs.
Whatever future we end up in, the likelihood of nefarious frontier models becomes more possible as costs drop.
To your point, as an individual, it would be critical to have your licenses well advertised that you are not responsible for loss associated with use of your code or app. We all bear the cost in this future.
Yeah, this is going to be a selling point of local models as the small memory options continue to expand.
Better tooling/reporting to demonstrate model degradation is needed though.
Even if there's zero malice in what these labs do, the reality is the local models are demonstrating that having the Uber Mensch of models isn't actually the "TAM" they forcast. The fact that a lot of people get along with models on consumer hardware scares them because their large models have a very small actual audience. Which means their future is closer to your utility closet than any magnificence of the singularity.
I have built some ridiculously cool stuff with free tier Gemini Pro handling online research/plans and Muse Glimmer 30B handling "fill in the middle" of Gemini plans; this with a single 64GB+24GB 3090, 2023-2024 i5 CPU (bought right before price fuckery)
Am talking about low level Linux kernel interfacing stuff, my own Google Opal like UI generator...still rough but it's machine state to bang into shape not a bridge. It works but needs the refinement common to early development
Imo due to recent job trope developments a lot of devs expect a git repo that feels like a 3-ring binder with their senior thesis and such tucked within
Local models inability to one shot such is being used as cope. Imo these models are more capable than we want to believe
This all reminds of 20 years ago when sysadmins refused to acknowledge the cloud would take off due to privacy concerns of businesses
Look how that went
Data center needs are inflated to try and capture all the business but future devices capable of optimized battery/telemetry aware local inference would satisfy most people
The data centers serve 3 groups; SaaS workers, data center landlords/CEO/shareholders, politicians being paid off by data center landlords/shareholders
The other billions on the planet do not directly benefit as much. Merely as a side effect of trickle down
I spent about a week as a side project trying to get qwen 2.5b running locally to be a job application filler bot with toolset. It was amazing to me that it was able to apply to about 65% of the jobs, based on certain issues that might arise, sucessfully with that toolset, when given the job app urls as the only starting point info (plus user job data/resume in json).
Of course the Sota models on the same custom harness got to about 99%, but I'm amazed at the performance of such a tiny public free model running on my 3080 laptop gpu. If it was actually finetuned for said task I imagine it could get to 90%+ completion.
Right, my point is quite clear: Even if their "total addressable market" is a real number (big if), that TAM is probably easily divided up by tiers of models, and a significant portion of those tiers is already covered by local models.
Of course, any given person might want their own navier-tokes agent swarm of a solution, but the number times you want a bot to fill out a resume rather than solve a millenium problem is absurdly lopsided.
So it's just kinda, ok guys, are you going to just try to hold us hostage with your magicaly super-duper-intelligence, or are you going to admit that having great models means humanity gets a few smart people to be even more useful for whatever they're working on.
The rest of us arn't going to even see that future to begin with, because food, it's not about how much food humanity grows; it's how it's distributed. Unless this AI stuff solves a problem _no one_ has solved for food, it's just going to be another power chair to fight about who gets to sit there, who has the right to sit there, or like I suggest, it's just a utility thats to be managed by the same governance all other utilties do and has the same problem thereof.
So like, boring stuff; not p(dooom)
Yeah, I'm also amazed how the conversation is always about 'coding', like these models are only good for that, and are going to replace all software engineers while leaving everyone else alone.
IMO these models are best at non-coding tasks. They of course have to use a computer, but from CAD generation, 3d Modeling, emails, legal and academic analysis, etc. these models are incredibly capable, and we are going to see a massive shift in the entire job market due to this, and ironically I suspect one of if not the most protected career as a result of this massive upcoming buildout will actually be software engineers.
Essentially I agree about not reducing them to just coding apps.
Am ultimately whittling everything down to Linux kernel + inference + data model so far captured from my current OS use.
Am ready to be done with syntactic programming of the machine. Rather than label the filesystem with context I can label entities in the viewport and those entities can happen to look like word processors not just video game characters.
An basically making a Blender-clone designed to browse and edit models that replaces the usual desktop metaphor.
Then I can noodle the virtual geometry, label it and understand under the hood its just a function factory mutating geometry of memory address truth table and display truth table.
The last gasp/push to rid ourselves of OOP is upon us; code objects of all kinds gonna be dead sooner than later.
Not sure I agree then software engineers will be insulated when everyone can do state management through visual edits. Maybe visual design work will still be a thing; the how to get machine into desired state is most certainly not going to be with visual syntax structure fir much longer
[dead]
How do you deal with Sam Altman pressuring you to give up your RAM?
when they demand you turn in your ram for national security?
Anyone with evals picking up a real signal? I've learned not to trust "vibes".
The only one I'm familiar with testing Opus 5.5 specifically over time is https://github.com/ninjahawk/livenerf and it's not showing anything.
We do, but it’s not definitive. We are building a battle system on top of OpenCode [1] and Opus 5.5 seems to have dropped since Nov 3rd.
If you calculate Elo only before 11/3 and compare to only after 11/3, the Elo is 52 points lower after. Unfortunately we are still in beta so this isn’t conclusive (confidence interval of -160 to +40).
If we do this calculation on all our models, Opus 5.5 is first in apparent drop. (Again, not definitive, but supports the conclusion.
[1]: https://agentduel.com/leaderboard
Lifenerf is really not sensitive enough.
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I made a CLI entropy generator project.
It's hard to tell, but I think they also pin the harness version? Updates to Claude Code can also have big effects on perf, and Claude Code updates a lot.
This is a good point. Importantly, Claude Code updates change the system prompt and that has both direct and very broad effects on the output. Livenerf hasn't changed the Claude CLI version in 2 months.
They also have this comment in getting started: "Pin the CLI. This is not optional: a Claude Code update changes the harness, and a changed harness looks exactly like a changed model. Turn off auto-updates and write down the version you're pinning"
Which seems both valid and wild. Valid that if you want to compare model performance you need to limit the variables.
Wild because as an end user I don't care what Anthropic changed to affect performance, I only care that Anthropic caused degraded performance.
It doesn't even matter if you pin them. Claude Code and Codex have "experiments" baked in and they can change your cohort remotely.
So your harness behavior can change without updating it.
Fun.
How do we know that they are not targeting this specific "benchmark" and serving it the same quality model?
> livenerf makes everything else deterministic: frozen prompts
One way to test my theory would be to run the benchmark yourself. (This catches if they are targeting this one account running the benchmark, have to say kind of unlikely...)
The other would be to see if something changes when you include this in your prompt: (this is the system prompt used by this test)
```You are completing a task that is graded automatically and exactly. Work carefully. Follow the output format in the task precisely: graders do not read anything outside the requested format.```
To control for changes caused by the addition of this extra prompt itself, we could try to add some other piece of text that is semantically the same in meaning. (wont be perfect but maybe its close?)
To isolate impacts further we can do the same test with other non-nerfed models. (we ask, is Opus 5.5 the only model that is suddenly doing a lot better after i add this magic string in my system prompt, or will gpt5.6/quen 3.8 etc get better too)
Exactly right. I feel like anyone with a comprehensive eval suite (companies running their own agents) should see this signal and it would be a well known fact by now if that were the case.
The only thing I could maybe see is that any changes that are being made are happening on the subscription tiers on not the api tiers, but even that seems like quite a stretch.
Like anyone who is building agents with a good eval suite would know, models have been pretty consistent and even small harness changes can impact performance considerably.
I follow this dude: https://x.com/bridgemindai/status/2110774897064681884
He has his "nerfbench" he calls it. Very useful and entertaining at least. :)
The last check was executed Oct 16: https://www.bridgebench.ai/nerf-bench
At least when I saw this thread I ran subset of my evals and therewasn’t any degradation. But I am using opencode with business copilot so it might be completely independent from average code + Claude subscription experience.
I wonder to what degree do they pattern match and route these tests to the "unnerfed" cluster...
This should be at the top. I'm so tired of people vaguely claiming models are "nerfed" when it's far more likely that your subjective experience has changed...
Definitely would be good to see more objective figures but benchmarks arent always great samples either. This one is also showing its not degraded but IDK methodology https://www.bridgebench.ai/nerf-bench/claude-opus-5-5
If there was a silent degradation then the recent chatter about suspected nerfing is what you would expect to see. Of course that's not evidence, but I dont think we can say it's far more likely that its all subjective.
I mean the basic intuition doesn't seem unfouded. Maybe people can confirm/deny this contention for compute. But Anthropic is compute constrained. Inference takes compute. Model development takes compute. Isn't it plausible, and even likely, that there will be periods of contention? Would Opus 5.5 performance just silently get worse if they gave it less compute, as opposed to just being slower or something? Or do we have information that suggests inference compute is affected by model development?
Monday night was my first session with the nerf and I can't say if the capability has diminished, but absolutely noticeable changes:
- claude used first person pronouns minimally before ("I","my") and suddenly like an average of every 3rd response has them.
- claude introspection'chatter' has increased like 5x over last week ("I'm doing this now. I will run this. I will check that."
- tool calls are like 6x more than what they were. last week it was <prompt>, thinking, tokens, work, response. Now it's prompt, thinking, introspection "i'll do this", tool call, tokens, introspective response, tool call, work, tokens, tool call, ..., response.
This one is not showing degraded performance, but that's only
> as of its latest test on Oct 16
Please stop gaslighting people. I'm tired of people claiming that my subjective experience has changed and not the observable behavior of the models.
Anthropic has confirmed that they make changes to the models, for example with the recent update to limit reverse engineering.
I think it's similar to how an artist's eye is ahead of their hand [1]. As you get more experience with the models you start to notice their shortcomings more and more. When a new model comes out, everything seems crazy good for a while, until you start to realize the weak points again. Benchmarks aren't everything, but people have created numerous benchmark trackers because of this phenomena and no one has caught degraded performance yet, other than this post catching that something changed (no connection as of yet to performance though)
[1] https://www.rensoart.com/why-your-eye-is-ahead-of-your-hand-...
Sorry but I have to disagree. Personal experience is subjective, I get it. But also like an artist, you develop a pretty intuitive sense for how things are working. If you do a lot of digital photography, you could immediately spot differences in camera settings just by glancing at the output pictures. Someone who works a lot with Claude Code or Opus 5.5 can reasonably develop a similar intuition.
With Opus 5.5 I (subjectively) observed decreased performance across a variety of projects from greenfield to mature, across a variety of tasks, and a variety of days. Across a variety of projects is evidence against the claim that this is merely a model struggling as the project grows. A variety of tasks suggests it's not just "oh you're doing something you wouldn't have done before" or "your expectations are just higher now". Over multiple days suggests it's not just my subjective mood that is changing.
No one catching degradation in benchmarks doesn't really prove anything besides Goodhardt's law. It could mean that the performance truly hasn't changed, or it could prove that Anthropic runs the same benchmarks when verifying the performance. Just like I can have a fully passing unit test suite while my entire project crashes and burns, the benchmarks are only good to the degree they accurately represent real usage. Which in my experience they don't.
And finally, in this case with Opus 5.5 there are people finding concrete evidence of the degradation by analyzing their Claude Code session history.
So to my main point, I wish people would stop trying to gaslight us or defend Anthropic. No one outside Anthropic truly knows what's going on with these models. We have some direct statements from Anthropic. We have tons of reports online that seem to indicate tons of users (subjectively) experiencing worse performance all around the same time, and we have our personal experiences of whether the models perform the same or not. So rather than assuming everyone with a bad experience just has bad expectations or is some sloppy vibe coder, maybe we can just say "it's unclear what's going on." (Not saying your comment is pejorative though)
What is most appalling is that a significant accidental degradation happened back in March, was explicitly confirmed by Anthropic, and when these conversations happen again, everyone forgets about this important prior incident.
A large group of people reported that Claude Code's capabilities were dramatically reduced. And we heard then what we heard today; we need objectivity, we're being paranoid.
But there was indeed a bug, a bug that excluded thinking tokens from being included in requests after 1 hour idle. Reasonable for an old request one time, but the bug was that thinking tokens continued to be excluded even if you started a /new session, unconditionally.
And now we can't see thinking tokens and look out for this issue.
It feels more like a crowded casino where someone asks "Is it just me, or is this table cursed?" and 50 people all run to agree.
Seriously. "Something changed" is not a "nerf".
The theory of expectations or our psychology adapting and seeing the flaws is also a credible hypothesis.
It seems to be an A/B testing experiment:
https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
This account claims Sonnet and Opus 5.6 are in A/B testing now. There is nerfing when they release a new version, for reallocating resources.
https://x.com/tokengremlin/status/2119783808773841101
how reliable is this twitter account?
i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
https://www.ueddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
And here I thought it was them taking away datacenter usage so they could drop a new model soon.
Do they ever test by giving the model more resources to see if they can give users more for the same price ? Just saying
They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.
You know what will never ab test your usage? Local ai. Just saying.
I tried it. Nothing local or open weights can touch Fable/Opus/Gemini for my use cases.
Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.
I'd argue that Qwen3.8-Flash-Next is basically at Opus 4.6/4.8 level from about 6 months ago.
Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?
> Maybe at Opus 4.6 level for the "easier" tasks.
I've got QFN making a C(sub/superset)->microcode compiler, writing it in OCaml. It's come up with optimizations that Claude didn't come up with (in the 4.6/4.8 era) when it was making the previous C-based implementation. The compiler generates microcode for an algorithmic state machine/transducer in addition to Verilog. I've been working with QFN on this for about 4 days now and it's pretty close to feature complete - yesterday it was able to compile our Game of Life C implementation and the microcode ran on a Tang Nano 9K FPGA board with LCD displaying the cells. I do not think this is an "easier" task.
What open-weight models can match Opus 4.x or GPT 5.4 performance at a reasonable throughput?
Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).
Agreed.
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
From experience, qwen3.8 27b with gsq/rco at iq3_xxs has good performance with little intelligence loss.
Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
Interesting, mine freezes the whole laptop:-)
Last year's state was not good enough for me, but this year's state next year would be great
I am also curious what exactly you tried. Depending on your hardware it might also be worthwhile to stay in-tune with Qwen 4 and GLM 5.5/6.0 leaks.
What it did you try, and what are your use cases? Genuinely curious.
Local AI is the way forward. Using someone else's harness and someone else's model is just borrowing a workflow that's liable and likely to change on a whim.
Yep, this.
And with Strata running at 50tok/s on my desktop is all I need. Would I like faster? Sure! But it's super useful in all sorts of tasks now.
100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
Assuming you built a top of the line PC a year or two ago or have a huge amount of money to drop on one now
I noticed Fable decline in quality a couple weeks ago. And I've noticed Opus drop over the last week, even after bumping to a default of Extra High reasoning for almost everything.
Tonight has been maddening with the kinds of absurd mistakes it's repeatedly making. It's so frustrating.
I'm the kind of person that wants something to be consistent and reliable. And Anthropic has been great, except for in this way. Which is enough of a recurring frustration (about every couple months this year) that I genuinely can't wait for other options to catch up. Then I'll be leaving as soon as I can.
I just don't know how long that will be.
I've been using Fable for a couple months as a relatively independent, long-horizon task executor and it worked well. About two weeks ago it completely stopped working. Like ~85% success rate down to ~10% in my workflows.
Totally possible that I'm just hallucinating/projecting/etc or my workflows somehow changed in terms of difficulty, but it's been kind of shocking.
I have the same exact experience. Working on one project, brainstorming. It was brilliant in July and August, like a seriously smarter, more experienced version of myself, reading in my mind, one step ahead of me, when pointed out a mistake it would remedy on the spot with the exact brilliant solution.
Now it's like talking to my hungover self, after 3 days bender, who just wanted to be left alone.
Same for me, especially the last couple of days. It was really frustrating and it felt like Claude was sick of working with me and just didn’t give a f*ck anymore
I guess there are experimenting with they can enshittify. But even if they undo this current change, all commercials model will eventually be enshittified.
what does "enshittify" mean in this context?
Silently switch the model you are using to a worse one.
Distilled as much as possible
Same thing for me, but the day Fable got lobotomised Opus 5.5 started actually working pretty well for me.
>I'm the kind of person that wants something to be consistent and reliable
And people wonder why open models are a big deal, and why some people love them so much, despite being 30% worse than the frontier.
Having control over something that writes most of your code is important.
And also the frontier has made everyone pay for a subscription to write code... that was a rare thing in the past now its common.
I read somewhere that the test might switch models mid conversation which stops the other from reading the previous model's reasoning. I don't know if that's true, but it sure feels like that quite often in the last days.
Codex itself displays this warning when your change the model mid-conversation.
I just cancelled my x20 anthropic sub after their last usage reduction shenanigans, and moved over to DeepSeek and opencode go. Yeah opus is a bit smarter, but not even close to justify the 20x price relative to DeepSeek.
> I'm the kind of person that wants something to be consistent and reliable.
Only local AI capable hardware and frontier open weight models can provide that.
Bumping to a default of extra high reasoning for almost everything seems ill-advised.
"More reasoning" != "better"
Oh don't worry, you can just switch to Fable 5.5 tomorrow, which is the smartest model so far (because the others are dumb now).
You definitely called it.
Haha, it was obvious, really.
Surprised they didn’t wait till after the IPO. Any guesses on how models might change then?
> Surprised they didn’t wait till after the IPO.
That's exactly why they did it. Nerf Opus 5.5, then drop Fable 5.5 just before IPO so it looks better next to its weaker predecessor.
> I'm the kind of person that wants something to be consistent and reliable
Surely everyone is that kind of person.
Who wants to pay for a service that doesn't work properly half the time?
Fable has most definitely been nerfed for the last few weeks, not to mention painfully slow.
I'm a pretty voracious 20x user, and OAI Biz Pro user; however I'm not an engineer by training, a previous tech adjacent dev team leader who always relied on pseudo code and broadening my horizons when dealing with my team. ( AdTech, DTC marketing )
So I'm AI pilled and in another career, and started to think, what if the problem is the inputs? They release Opus X.X and at first it's a rush of their best users, best use cases, upgraded lanes. As weeks go on, adoption increases, cruft uses start to poison the model? Is this how the ongoing evolution works? I'd love to get Sonnet 5.5 and Opus for review in a place where I could reliably daily drive for X months / Y years etc.
According to our measurements this isn't true. We re-ran Opus 5.5 today on the benchmark we ran in September, with the same prompt and settings. Before and after, side by side: https://playcode.io/blog/macbook-svg-benchmark#reruns
Max, streamed (128K cap): 128,000 tokens, $2.56, empty file, in September and today.
Max, Batch API (300K): 255,957 tokens for $2.56 in September, 243,707 for $2.44 today. The two drawings: https://playcode.io/marketing/assets/static/claude-opus-5-5-... and https://playcode.io/marketing/assets/static/claude-opus-5-5-...
Xhigh: 86,096 tokens for $1.72 in September, 107,333 for $2.15 today.
The Batch API bills half price. Max ran out of the 128K ceiling again, to the token. Given 300K, it drew the same kind of laptop with 5% fewer tokens. Xhigh used 25% more and drew the same kind of laptop. That's inside the noise: 17 identical calls at high ranged from 17,757 to 59,705 tokens.
The model is still the best of everything we've tried. Most likely the next best would be Fable 5.2. But we don't like Fable because essentially the result is the same, just the price is higher. Its medium and high are close to what Opus 5.5 Max had, but the price is higher.
(I run Playcode, an AI app builder. We benchmark every model before we add it.)
Could they be cheating when a benchmark is detected? Would that not be an obvious feature?
It seems like this would be both a trivial and obvious feature if they were in fact nerfing
Thank you for taking the time (and tokens) to check this!
API != Claude
Regressions are often caused by changes to the harness.
(My experience is that overall these changes have been huge improvements, but the occasional bug or mistake does happen.)
I noticed the same, and posted about it three days ago.
Opus is suddenly fumbling basic tool calls and failing to finish tasks even at high, where I had been permanently using it on low effort with success since it came out. It is very frustrating.
My guess is they quantized it and/or reallocated DC capacity to train a new model, this has happened before.
> It is very frustrating.
Not just frustrating, it can also be dangerous. What if you trust Claude with certain security-sensitive things (because experience), and suddenly you cannot trust it anymore like that?
You shouldnt do that in the first place
This argument would be much more convincing if Anthropic wasn't constantly trying to sell everyone on the fact that you should.
It’s happened before and it will happen again. At least now you know.
And knowing is half the battle!
Then you should have evals.
> What if you trust Claude with certain security-sensitive things
then $corp should remove your access to these security-sensitive things ;)
Maybe don't trust something with non-deterministic output for critical level security?
*chaotic
It is deterministic, if we're being pedantic
If we're being pedantic, it may be deterministic from the point of view of Anthropic. They can guarantee that my prompt X results in actual prompt X+Y (Y being all the settings they control), which would hopefully result in deterministic output Z. From my point of view, Y may as well be random, resulting in non-deterministic output Z for my input X.
It really though?? Even if you set temperature to 0 I saw research showing that the scheduling of the order of operations in the GPU firmware itself can change trailing decimals places and affect the output token
I'm aware operations can technically finish at different rates throughout all of a GPU's cores, but I am not exactly sure that GPU firmware would change the outcome of a calculation (unless there's something specifically in the firmware for it)
I don't know a lot about it tbh so I probably mislabeled it as a firmware issue
It is not deterministic.
Everything might be deterministic if we are being super-pedantic.
Instructions unclear; revoked all human access.
Right lol, i didn't know we'd built a machine that was capable of getting critical level security deterministically correct. That's not me, at least...
It literally could have been rolled out in the middle of that brain surgery experiment on a mouse. Woops.
Anyone who chooses a cloud model instead of local for medical use has a damaged brain.
You don't need to worry about this. If you've outsourced your thinking to a cloud model, it's going to take your job soon enough. I hear layoffs are popular around the holidays ;)
It seems to me the answer is in your question.
And it’s here: https://www.anthropic.com/claude-fable-and-mythos-5-2
Multiple instances of failing a Bash tool, calling bash (lowercase) and getting rejected and even reprimanded by the API "No such tool, call Bash instead".
Killing it's own shell, multiple, multiple times in the same session. I'm sure I saw a phrase "I did it again".
It's so shit that it must be damage control. The alternative is to limit usage and that'd affect the IPO even more.
[dead]
I noticed its been trying to drag out the conversation on every topic. Maxxing token consumption for the dollars instead of staying on point. Makes sense for them but its frustrating for me as a user. I suspect this is going to be the AI version of enshittification. Now its only missing some badly needed in-stream ads.
It blows my mind how people still build their workflows around depending on services like these, with how easy it is to just self host and forget about all of this. You can buy a GPU that can run Qwen 3.8 flash next for about 9 months worth of the highest tier Claude Code subscription. You'll never have to worry about token limits, the model changing out from under you, getting banned, etc.
It's because the frontier models are superior and much faster.
>much faster.
This really depends on your infrastructure. In this case it all depends on your hardware, and there is no secret sauce, you can buy the same GPUs or better than the big players are using. Self-hostef GLM in our company is faster than any commercial model. Of course the hardware was not cheap, but completely manageable for a medium or large company.
Speed can mean two things: rate of token generation or time to finish a task. What maters it the later and a model that takes a faster path to the result and uses fewer tokens because of that comes out ahead in practice.
> you can buy the same GPUs or better than the big players are using
You can not buy an H100.
Superior? Sure. But do people need the capabilities beyond what can be reasonably achieved with local models? I think the people that actually sit in that gap are vanishingly small, and if your title contains "Engineer" or "Developer" or something like that, you are not one of those people (at least for doing your job)
As for faster, maybe? It depends on the metrics and what you're doing with it. If you mean TPS, no, I get much higher TPS and even faster TTFT on my local setup than frontier models. If you mean time to complete tasks... maybe. If you're throwing whole projects at it to vibecode, yeah, a frontier model will dump slop so fast it'll make your head spin. If you're doing actual methodical development, like doing spec-driven development and actually reading the specs before spinning off agents to implement them, rather than just having another agent review it and yolo, you'll find that frontier models don't really push the needle that much, and you'll be making high-quality projects.
How much did you spend on your local setup out of curiosity?
Also, yes, I do need the capabilities beyond what can be reasonably achieved with local models because threat actors are also using frontier models.
>> But do people need the capabilities beyond what can be reasonably achieved with local models?
Yes? If you're building anything reasonably complex, it is a ton of work to keep everything in your head so that different parts of the system work seamlessly and work you do in one area does not adversely affect another area.
Every hour baby-sitting a local model can be extremely expensive in billable hours.
Is there a model that doesn't need babysitting? Half my day is spent untangling BS that Claude and Codex hallucinate. Particularly around hallucinated product requirements. Its still faster than hand-writing code but only just. I'm seriously considering just going back to the early copilot days where I lead coding and just let it try to auto-correct. That was a great balance of AI enhancing my speed and workflows without removing any confidence in what was being shipped.
give your models a more organized context/harness if they are hallucinating product requirements... and I'm not talking about some hulking context framework. just organize your documentation in a canonical way. This product requirements hallucination is something that would bite me constantly if...they ever actually hallucinated them (and they don't, ever). This is most certainly user error.
You're untangling hallucinated BS from Opus 5.5/Fable?
All the time. All the frontier models love to take a requirement and bend it into a need to create an obscene amount of overhead. I needed to add a permissions for a user type. Instead of adding the type to the existing list of permitted types it created a bespoke experience for that user type complete with a full testing suite. This is without diving into the absolute insane amount of no signal tests that these things generate. It once decided the privacy policy should have its own testing suite, which basically amounted to 50 different regex tests looking for "key" words and phrases, and snuck that into a PR for something tangentially related. They love to burn tokens on creating useless shit and it happens on the daily.
We are literally in this thread because so-called frontier SOTA Claude models are shitting the bed and require babysitting. So this argument is essentially moot.
I might not need them but I'll sure as hell take them if they are available
What’s your local setup? I’d love to give it a spin; lack of 5 hours limits is much needed.
Qwen 3.8 (and maybe soon 4) series models are probably the best right now for local AI. Specifically, for lower end hardware like 8gb GPUs you might find a lot of use in a Qwen 3.8 variant like Underdog Saluki 27B.
They are faster, but the distance from superior is changing daily. Naturally, things like purposefully kneecapping the product does not help.
nobody with the competence to judge the quality difference between frontier models and local models would ever, in a million years, use local models
Today this is absolutely true. Local models have gotten very good, but they aren't there yet. The cost of a Max subscription or even API pricing is overwhelmingly worth it for most users, even if you ignore the enormous capital costs of running the competitive models locally.
But the in a million years bit, even taken as rhetorical bombast, is of course silly. The difference between local models and the frontier models is ever shrinking. Extrapolating out and eventually, sooner rather than later, the difference is going to be a wash.
Unless all of the vendors stop progressing open local models, the future is clearly there. It's why I think AI is important and valuable, but also the current leaders have a dubious financial path forward.
It's an easily reproducible technology, clearly...
He's pretending like there's any magic to claude or the others, there isn't
Why not? There are a lot of tasks that a local model can do cheaper.
In a workplace environment I don't think "cheaper" is necessarily the thing to optimize for. Subscription models save you a lot of time (and money) by getting things right faster, solving bigger problems, etc. A cheaper model just consumes more human time.
But with a local model with essentially free tokens, I can set it to work on tasks in the background without having to worry about any usage limits. Deferring simple bug fix tickets or other low complexity to an essentially unattended local model works just fine. And if the result occasionally fails review, it still didn't burn paid usage credits.
A local model tops 60 tokens/s max. Running it on the background means not being able to use on the task you are working on. When working on local hardware you cannot do concurrent requests. In deepseek for example you can have 100 agents running each getting at 120 tokens/s easily. In local models you are limited by your own infrastructure.
I do concurrent requests just fine and they have no noticeable performance impact. Not all local hardware is equal. Also, humans have to sleep. Agents don't, so processing stuff over night and reviewing in the morning just gets more utilization out of what is already there.
With cloud agents I also process stuff overnight. But my gpu is limited. If I want to OCR a high number of documents or vectorize data, I can't run an LLM at the same time. I use remote models and use my compute for running commands on my desktop machine.
You seem really upset that people do things differently than you. May I suggest trying Tetris to help you feel better? https://play.tetris.com
This only holds if you do not value your time meaningfully.
Why? The part that takes the most time in this process happens while I sleep.
Enjoy renting everything you depend on!
I sure do. And 2-4 years in the future when this insanity is over I will buy and own local llm hardware. But certainly not now with those prices and speed of progress that obsoletes everything in a year or two.
I made some assumptions and was foolish. Sadly not a new tale. But still in context I do think for every You there are several other Not Yous who are planning infrastructure who aren’t keeping an eye on the local escape hatch.
Compared to Opus local models Opus 5.5 is far superior, also for something like a dgx spark at 5k that would be 25 months ignoring power costs if you chose the $200 plan.
And that’s looking at a large model, to host the equivalent sized model as Opus 5.5 you’d need more than a single dgx spark. A fairer comparison would be to Open models hosted remotely or comparable weak models like Luna which you would get away with a $100 plan or less. That also doesn’t account for token prices continuing to go down for the same level of intelligence.
Are token prices going down?
It has been trending downwards. See recent Anthropic and OpenAI price changes.
But even if that was not true, the more important metric, cost per task, has definitely gone down.
But newer models also use more tokens. Is this really evaluated anywhere properly
Usually, newer use less. Outside of the crowbar approach on max mode, occasionally.
technically maybe but providers are known to tune/nerf plans to that effect (OP case in point;)) so factually idk
Yes, by a lot. You can get opus 4 level of intelligence like 100 times cheaper now than at premiere.
it is believed that opus5.5 needs 8x h200 to run (~1t vram)
a dgx spark could never
What GPU can you buy that will run a 180B model for $1800?
Qwen 3.8 Flash Next is 125B with 6B active, which with quantizing and system memory offloading (strata) runs nicely on 32 GB cards (I have actually run it on as low as a 16 GB GPU and 16 GB of system ram, but I don't recommend it for serious work). I was running it on a single r9700, but now I'm using 2 r9700s because I'm splitting my server with a friend, and they bought one as well.
On the single r9700 I was able to run q2 with 4 slots of 256k, and I think it was about 40 - 50 GB of system memory offloading? Got about 60 - 80 TPS per slot depending on concurrent use. With the two r9700s we are running q3 with 4 slots of 256k, using about 100 GB of system memory offloading, with mtp enabled. Getting about 70 - 100 TPS now.
You asking wrong question. You need to ask what tks/seconds it's gomna run at.
There are plenty of ways to run ds flash and qwen flash on single cheap gpu, but speeds are quite bad or contex windows are small.
i dont understand how people keep comparing local models with frontier models, they dont belong in the same discussion especially for coding usecases. they solve different problems under different constraints.
I don't understand what do you mean. They both solve the same problem. Some models solve it better, some worse. And in some cases you can only use local models, so they solve more problems.
"Frontier" (closed) models are better by definition, but if frontier models were good enough for you half a year ago, open models are good enough now.
This whole recent "good enough" phrase that is being thrown around more and more is completely asinine.
"good enough" leaves so much on the table...and wastes so much of your time. I value my time and attention
Indeed. Local models remain rock solid, while cloud models are a shitshow of stability.
What is the alternative? Local models? I do not think so. Maybe Kimi K3 (still not local) as this model can be served by multiple providers? However Kimi K3 sits between Sonnet/ Sol and Astra/Opus so still not the same level.
Lots of models are being served by many providers. Even Anthropic models can be served by Azure. Check for example https://www.requesty.ai/models
No. That is just Anthropic allowing US based companies to host it in the US and EU. You can buy a few B300 and run Kimi in your data center as a company. Not possible with frontier models from OpenAI and Anthropic.
I can buy a GPU that can run Qwen 3.8 Flash Next for 9 months worth of the highest tier Claude Code subscription, but then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
> then I'm running Qwen 3.8 Flash Next when I could have had nine months of unlimited Opus 5.5.
By choice, maybe. New models release faster than every nine months. And, evidently, you won't be running nine months of some fixed meaning of "Opus 5.5". What that means changes over time, out of your control, and without ever even being officially told.
Sure, but, even when it degrades, it's better than Qwen 3.8 Flash Next.
Now, if GPUs didn't cost an arm and a leg, the economics would have been different.
20 euro claude sub is too good compared to a 6-10k hardware purchase to get something thats not even as good. Once the enshittification starts on full force it will be a different situation.
The value of a Claude sub has been dropping by like 50% every two weeks at this point due to usage reductions.
I honestly thought usage was going up. Now, I dont use LLMs nearly as much as a lot of people here, but I used to hit my limit very easily, and now I haven't hit it in wmquite some time.
I don't know how you thought that when real usage is down 10x since 2 months ago, they publicly announced announced several usage decreases and related issues, and its a constant point of discussion/criticism online.
I thought that because I used to hit limits and now I do not.
Wow that's cool
Voluntarily downgrading to Sonnet 4.5 level performance sure is one way of avoiding imagined degradation in Opus 5.5...
Have you ever heard someone say they did so and regretted it?
First time I hear someone praise Qwen 3.8 Flash Next specifically.
Personally I was forced to use only GPT-5.6 Sol at work instead of Opus 5.5, and it was so much worse.
Benchmarks that control for benchmaxxing indicate all open-weight models behind 5.6 Sol and roughly at the level of GPT-5.5.
And that's for the latest large ones, much less local Qwen 3.8 Flash Next.
I would regret such a downgrade.
i haven’t heard anyone say they downgraded their car to the children’s toy power wheels and regretted it. all the same, i’m confident that it’s a bad idea
Why lease a Ferrari when you can buy a Corolla.
This, but unironically.
You two actually summarized the whole discussion beautifully.
Why indeed. (The Ferrari's always in the shop, and the Toyota gets great mileage, and you need to commute.)
If you’re commuting then sure, but most people are racing.
If the car's in the shop, you're not winning any races.
> You can buy a GPU that can run Qwen 3.8 flash next for about 9 months worth of the highest tier Claude Code subscription.
which GPU?
also, so, 90 months of my subscription? And I also would need to buy new RAM and other parts of a computer?
Problem is that everyone loves to pretend their CRUD SaaS is special and requires the best of everything, so they 'need' the absolute frontier with as many tok/s as money can buy.
Yeah but what about your duty to consume and be part of the economy?
can u tell me the specs or hardware requirements to run qwen3.8 flash
which GPU?
R9700. I walked into Microcenter on Hallow and picked one up for $1900.
I was also using a 7900 XTX before that which was reasonably capable for single user usage, but I share my server with a friend, and the 9700 is a much better fit. Also nice for doing sub agent workflows that I couldn't really do before.
[dead]
As one user pointed out:
> I noticed this too as did many others. Had to bump my default reasoning but it’s not the same. Seems to often happen before new model drops
I heard that Qwen4 was dropping some time soon, I wonder if Anthropic is trying to announce a new model immediately after Qwen4 - both for their IPO, and to kill the competition.
This has happened to consistently across OpenAI and Anthropic models that I can't believe people are still denying it. Obviously they're silently quantising their models or making other efficiency changes. If they can save 40% RAM and only hurt intelligence by a few percent in their benchmarks, they're going to do that. Unfortunately their benchmarks are obviously not representative of the real world, and these downgrades are very noticeable for people who rely on these models for work.
Of all the reasons to look at local inference, this is the one which pushes me in that direction the most. I want to rely on a model for consistency in my workflows. If I wake up on Monday and suddenly it's refactoring things it built on Friday, that's a huge problem. That makes it unreliable.
> I can't believe people are still denying it
Probably because those "nerfs" do not show on sites that track model performance over time. Sometimes Anthropic/OpenAI mess stuff up in their harness, but usually not in the models themselves.
It's easy to imagine they can detect when they're being profiled and route accordingly. Paranoia or not, it's possible.
I guess the question really is: is this malice (trying to trick people into thinking new model is better by nerfing old one), or is this capacity constraints, where they obviously need to make room in their infra for inference of the new model by shoving the other models into a smaller box?
Little of column A, little of column B.
These machines are inherently unreliable.
People are denying it because it's simply not true and for sure has not happened consistently.
In my own work I have not once noticed a degradation where I could have said with any certainty that it's not within usual deviation or me having a bad day.
Conspicuously the model seems to be especially dumb and annoying when I did not get a lot of sleep the previous night.
When OpenAI and Anthropic engineers post here on HN that they do not quantize models and silently replace checkpoints in the API, I believe them.
I saw nothing that convinced me the people who are constantly making these complaints don't just mistake any suboptimal result for changes in the model.
I've "called" the last 3 out of 4 new model releases. Maybe my account it perma-tagged as one that gets downgraded to re-allocate resources before a release but about 5-7 days before a new model Claude suddenly gets dumber and dumber on executing canned skills that I've used for 6+ months with no issues. The cynic in me thinks they are dumbing down models right before a release so the new model seems much smarter.
It’s also true that people that perform their own evals see wide variance uncorrelated with “nerfing”. Human cognitive bias makes them only see that variance at specific times and then they start projecting.
Everyone really should keep a few tasks around that are indicative of the work they do and run them a few times with the same model and a few times across model families and when new models drop.
It’s extremely informative to see with your own eyes the variance that always exists. Even if you can’t afford statistical relevance it’s valuable.
Yes I have similar anecdotal experiences - it gets dumb before a new model. What I imagine in my head is Anthropic using all their GPUs to polish off the next model and they need to borrow GPU cycles so they cut down the thinking.
Token use exploded a week or two ago. I worked for a few hours and hit my 5 hour limit. Did only some simple bug fix. Somethings definitely broken.
This is the #1 thing i HATE with LLMs. They are not predictable like a compiler is.
It's not LLMs! Its their providers! If you self-hosted, you'd observe the slow down but you'd get same quality.
Weird! So you're claiming local models somehow are deterministic? Amazing.
All models are deterministic.
There is random input to make the models evaluate differently every time, but without that (or if you fix it) then the model will always give the same output with the same input.
There's no somehow to it, it's actually surprisingly hard to make a computer do something in a non-deterministic fashion.
I was being sarcastic. LLM output is not deterministic (at least not in any meaningful way beyond the fact that everything in the universe is deterministic).
I was being pedantic.
> LLM output is not deterministic.
Correct, but that's because of added random inputs not because the model itself isn't deterministic. Put in the same random seeds into your model runtime and you'll get the same output every time.
> everything in the universe is deterministic
Again, mostly being pedantic, but not everything is deterministic. Radioactive decay is an example of a phenomena in the universe which is non-deterministic. Most things are, not all things.
No, that's not what I am claiming.
Damn, do I hate such manipulating fallacies.
It's probably not token usage, but a flexible 5 hour limit. There's a reason why they only state it in percentages.
Nope, the issue was not the 5 h limit, but that the 5 hour limit was gone in some minutes and that this burned trough 50% of my weeks fable limit. Before I could do roughly one full 5h session (fable till max) per day.
They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.
Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.
My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.
I replied to your neighbor https://news.ycombinator.com/item?id=50432889
> we needed half a dozen lines to make
6 lines per model? Can't you put that 6 lines in a function/macro/decorator/library so you only have to call it?
I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.
Order of operations affects the result in floating point, even moreso at lower precision. GPUs are massively parallel devices with many different sources of latency, so order of operations is not guaranteed for things like dot products, where many threads contribute to the result. Consequently, you can get non-deterministic outputs even though nothing else has changed.
[dead]
Some teams I know started moving all linting and validation into LLMs and markdown files, the reasoning was that they asked agent to review changes that were later found to be buggy and agents flagged these bugs. And that’s great but my immediate worry was that: it finds it now, but will you check every new model for all these bugs? And what if you need to switch providers? Or your provider silently tweaks the model to improve their margins? The answer was “well also build deterministic linters for all these problems”, but you know that’s just wishful thinking - if you can get 99.9% done with just a little bit of fuzzy text and the next 0.1% is a lot of work it won’t happen. So you have model behaviour lock-in that will bite you suddenly when you least expect it. Or maybe LLMs will just keep getting better and cheaper? It’s all a casino right now.
> Some teams I know started moving all linting and validation into LLMs and markdown files
Sounds like those teams need better leadership. Deterministic tools should always be used along side LLMs to improve the results and reduce costs.
Every LLM company must keep getting better and cheaper to compete. If you game out them running low on money, none of the outcomes actually result in you spending more or getting worse models, just switching to another one.
If you have 99.9% already - that's better than any pre-LLM deterministic linter.
This makes no sense. Even if you're all-in on LLMs why not just ask the LLMs to interface with the static analyzers. Fix the bugs, add new rules, run the commands you always forget, etc.
It's like saying "save this json as .txt but using valid json syntax." I mean maybe that will work most of the time but WHY?!
- 1 week before releasing new model, they nerf the old. Probably to reserve compute for the new release. (insert your evil corp motive here). This is the third time I have seen this now.
- They run A/B tests continually. Hence the "how is Claude doing: good/okay/bad" question.
- They try to shrink the model while keeping performance, using quants, etc. they keep the name though. Opus at the first, is often not the same Opus a month later.
- My friend works at a top 100 spender on Anthropic. He complained about nerfing to his point-of-contact at Anthropic, and they did some setting change and it didn't happen anymore.
- He, European, says the models can get worse around 4pm, which is when America gets to work
I'm surprised why folks are either all in on frontier subs or self hosting on their own hardware.
To me it seems there should be a very viable middle ground of paying for open model hosting by a trusted provider - unless I'm only hearing from 24/7 vibe coders
I also try to go that route. Currently exploring Kimi hosted by Infomaniak.
I only see Kimi2.6, is that what you mean?
With open weights, YOU get to decide when the model changes.
Not true at all, there's much more to a model than its weights.
Like?
Thinking budgets…
In the context of this question, that's still something you as the user control.
Specifically
> YOU get to decide when the model changes
Yeah obviously but you can still control it.
You can still control it if you can trivially spin up a node with eight B200s and 2TB of memory to run your Deepseek v4 instance. Nobody on a subsidized plans is doing that. Local models that fit 128GB memory don't come close even to a "nerfed" frontier model.
Actually Qwen3.8-flash-next comes very close to a "nerfed frontier model" the way I host it locally at 4.75bpw with int8 kv. It is pretty much equivalent to opus 4.6 up to 4.8 on coding. What it lacks is some factual knowledge - this is to be expected, it's a far smaller model. But if you run the kind of coding tasks where you have to spec them in high detail anyway (large very custom environment) this model is very usable.
I run it on a 6*rtx3090 consumer PC (no threadripper etc sadly) with very constrained pcie topology (4 cards sit on 2 lanes basically) and i still get 7k prefill and 50t/s single user or 200t/s 7 users. All at full 262k contexts of course and with 96GB of ram kv cache that means I can easily have 10+ Claude code sessions open with 1-7 generating at any time.
I've been the person that said, before opus 4.6 came out that this AI stuff is absolute crap for programming. Then it suddenly wasn't and I was using it many hours each day.
Local AI was the same. It was all crap. Models like GLM5.3 Flash that are many times larger than that qwen, lots of people are oooing and ahhing over I found way too inferior. Previous DeepSeek V4.1 Flash 0731 was atrocious. They benchmaxed it to hell, but at least it was small.
The current model of same name but release 1017 and double the size is currently the best open source model IMO. It even beats my self hosted Qwen3.8-flash-next.
I've tried hosting that DeepSeek model, but even adding another rtx3090 for a total of 7 cards and my 192gb of ram bought when it cost under $1000 I get only about 25t/s decode and few hundred prefill. That is far too slow for me to use.
But, I believe once this pyramid scheme of circular financing finally bursts there will be a glut of hardware on the market where people like me will be able to buy equipment to run 750B models like we now run 120B.
I've been very impressed by it and recommended it to everyone. 32gb ram, 12gb 3080ti, and I run 3.8 flash next coder (half the experts ripped out, leaving in coding) int8 kv 128k context at 35-55tk/s and I agree with the opus comparison; I personally say 4.7 is what it seems like the most.
Truly insane times
I read stuff like this and just think that if these open models work well enough financially you'd still be better off using them via OpenRouter or some other hoster.
Local is not worth buying hardware for, it's only good to run on the stuff you already own. As soon as you want to do work you are getting screwed almost as hard as people in the 90s did with $2000 pcs that went out of date in under 6 months. Just pay 20 bucks and use the frontier models, they have better harnesses and features than we have access to anyway.
Just curious, did you put all these cards in a server or desktop case ? and what kind of motherboard ?
Also, how much power does that consume at full tilt ?
For Deepseek V4 Flash (not 4.1), you can use two 128GB unified memory boxes and run it at the official weights (mostly 4bit) with lots of context.
That's around 1% of the price of 8x B200. At current prices, it's as low as $6200. At last year's prices that's $3700 or so.
you can run Qwen3.8-27b with a single GPU.
Definitely a significant change. I noticed it in just daily work. Quality of work down, mistakes up, repetitive errors in scripts - rare earlier, now a few every session.
Definitely noticed some anecdotal declines this past week or so. Missing steps it didn't previously, needing reminders of tool calls and skills that it should auto apply.
Lots of "this is a potential issue, I didn't check into it though" which is super frustrating to see. The model just points out major flaws in logic or barriers but goes no further than a comment on it.
I've got a Qwen coder model and a flash next model locally now, so it's not the end of the world. But it does need to be remedied - are we seeing them prep for a new model deploy when this happens? Or are they experimenting with leaner hardware to drive down costs ahead of IPO?
I’ve been experiencing unreliable performance for a while and can’t imagine this is solely due to new model training.
It does seem like Anthropic is experimenting with how much less they can offer users. I also get the feeling they are judging what resources a task deserves and dumbing things down accordingly. I’m seeing lazy responses and skipped steps more frequently for non-technical queries.
It’s definitely possible the average user hasn’t noticed, unfortunately.
I moved from Claude to Codex and am happier. The prose is less grating and results comparable. As much as I don't like Altman I am tired of Anthropic's antics.
I tried this and while the prose is indeed snappier I find the code quality of Codex truly horrific. It's dense and uninterpretable. I almost wonder if they have made it purposefully opaque so that it's harder to resume work without Codex once you're using it agentically. Claude is still much better at writing quality code that is fit for human consumption.
I don't think Astra's code is quite as bad as you say, but yeah while it's much better at non-annoying prose, the code definitely isn't quite as good as Opus 5.5.
Currently using GPT-6.1 Sol Medium/High for coding and I'm quite satisfied. I do have clear instructions and style guidelines though.
Same.
I'm so happy to not look at bogus jargon that Claude made up.
That largely ended with Opus 5.5.
Not for me.
Um., Codex is now suffering from the same thing afaict. It is almost as if having a duopoly is a bad thing for consumers.
... is it?
I had a baseline evaluation from before Nov 3, so I re-ran it this morning.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
People, refuse to be A/B tested!
If you paid for X but are now receiving Y, you should be entitled to get your money back.
Every time I consider going back to Claude, I remember that mysterious week in February when Claude was failing to do even the simplest things. Things were so bad that BigPickle was correcting errors Sonnet was making.
And let's not forget the king of A/B testing here. Google has no issues breaking products for months just to test some ridiculous idea.These people should be publicly fired.
> Every time I consider going back to Claude, I remember that mysterious week in February when Claude was failing to do even the simplest things
100% agree. I don't care how many reddit bots i see praising Claude, I bought max once wanting to see what the fuss was about, and the next day they broke the product and it was really embittering
Please say more about your Google reference.
Google is known to make user facing A/B testing whenever they feel like it.
They can one day decide to rearrange the Gmail interface to absurd levels by removing the TO field or replacing everything with a big Gemini button. Then push it out without any internal testing to half of the planet, and suddenly you are doing IT support for grandma who is convinced North Korean spies have hacked into her email...
I guess it's just capacity problems. They probably have a fixed amount of compute and they need to make space for new models that will be released. So they quantize existing models e.g. in half so that can use 50% of current capacity for new models. They have such massive scale that you just cannot dynamically get that amount of new compute online so they need to work with what they have. No provider has thousands of GPUs idle.
They don't do that. They've said as much and there is zero evidence to support it.
Under load requests queue and take longer.
There are a couple documented things where it has degraded: in late 2025 or so when they transitioned to TPUs, they had a bug in the compiler affecting sampling logits in a cheaper approximate way than GPUs and that caused a big degradation. They never told users "sometimes we'll sample logits in a cheaper approximate way," but the main issue was the bug and not really an intentional thing at least from how the characterized it.
TPUs only made up a small percentage of the fleet, but, exacerbating that bug, if you got routed to a TPU instance at some point in your chat then another bug kept you routed there for the remainder of the chat.
The second one I'm aware of is earlier in 2026 they decided to save usage by stripping out all thinking tokens if you left your chat idle for one hour (cache timeout) and then came back to it, causing massive model performance degradations. This one also made me very skeptical of Chinese distillation, until the reports of how you could use Haiku to extract the raw reasoning of the larger models.
They fixed that, but recently they switched to doing auto-compactions for long running chats in anticipation of them going stale in cache. I think that one is usually what you want anyway and there is at least a way to recover to the uncompacted chat if you really need it, but it might also be a source of some degradation people are seeing now.
Yes, I'm aware that there have been a number of isolated platform and harness regressions (bugs and behavior changes).
But the model NERF conspiracies are bunk.
I think the bug one wasn't bad intent, but it was a fact that even without the bug you got different behavior with the approximate logit sampling on TPUs. I say that was jut a minor platform optimization though and wouldn't be too upset about them not disclosing it.
But the harness one was not just a random harness regression; they were getting a huge influx of users and pretty clearly did it to save capacity. Not a great thing when you might be relying on attention to those thinking traces to reduce the risk of prompt injection etc. and it could have gone beyond major task degradation into a safety/security issue.
Yeah, perhaps but they are also trying to make the harnesses more efficient so they are competitive with their competitors on efficiency. Reducing token use can also reduce context size and improve aligned accuracy in general..
Through the lens of a conspiracy theory this could be viewed narrowly as "cheating" their users to reduce load.
But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
> But there are other motivations for the constant work going on in the harnesses that paint a fuller picture.
Anthropic publishes extremely detailed changelogs for Claude Code with each release, yet this one wasn't mentioned when introduced on March 26, and it was right around a huge user influx causing capacity issues.
This is what I've been wondering. I mean we all know they are compute constrained so I get that intuition to a point.
However, how would less compute result in degraded performance as opposed to more latency? Like is amount of compute something you can actually tune to control for latency? Wouldn't that imply they are actually serving requests in other models? If I run a very large model locally on a shit computer vs. something maxed out, it's the latency that differs.
The major providers like Azure along with OpenAI and Anthropic have the concept of "priority processing" which comes with service level guarantees on latency and token throughput. It's more expensive.
They basically dynamically allocate compute into this pool from the general pool to meet demand and SLAs. If demand for priority processing spikes latency in the general pool spikes.
Almost all subscriptions use general pool. That's why it gets really slow sometimes. Unless you use "fast" which acts like priority processing and is super expensive.
So that's part of how they balance compute based on demand and fund the "excess" capacity. This is a bit reductive and basic, but it's all the usual Cloud on demand, reserved, and spot capacity techniques at play.
Thank you for the information but I'm not sure you understand my question. I'm not surprise to hear these details on how they control for latency.
Im asking on how lower compute could translate to worse model performance. That's the implicit claim being made when people are saying Anthropic nerfs a model (result is bad, not necessarily slow). The only way I can think that could happen is if they actually swapped Opus 5.5 out for a worse model that takes less resources to do inference.
Yes, that's what they are claiming; that the full(er) precision models get heavily quantized and/or distilled to lesser, "nerfed" models and requests get routed to them based on load, time since release, time to new model release, and reading chicken bones.
There is no proof, the frontier labs deny it, and certainly it's not happening on hosts like Azure, Amazon, and etc.. The claim is that it's "trivial" to detect observation, so they serve the normal model if you try to measure it..
Basically the conspricy theory is:
* OpenAI/Anthropic nerf their models regularly to reduce compute demand
* It's impossible to measure and thus prove
I've seen big drops in Opus 5.5 (on high/x-high) & Fable. Both in terms of following instructions / workflows execution and overall code quality. Also the token burn-rate increased a lot, I've reached limits of 3 Ultra subscriptions during last week & had to spend 1.5k on API usage during the weekend just to be frustrated with the stupidity of the output.
Weirdly enough, Sonnet quality seems unchanged.
> & had to spend 1.5k on API usage
You rewarded Anthropic's failure with more dollar spend. I get why you did it but...
For me I've noticed the pattern of decline happening on Tuesday and Wednesdays. I find myself having to have Opus redo many steps and making a much larger number of errors on those days. I am able to be more productive on Thursdays just as my weekly reset approaches on Friday at 1 pm local time.
I don't have measured data aside from myself noticing my frustration level with Claude jumping on these days frequently over the past several months.
It's just a new summarizer, the model and performance itself is unchanged.
+1, this tells nothing about the quality. Or even whether this is an upgrade or downgrade.
https://XNotMe.com/shanemelaugh/status/2119192075505893695
Yes. That was the weekend my token usage blew up and used up my weekly allowance in a matter of hours. It was so stark.
After I figured out, like this chap on X, that it wasn't ME, I diversified my model usage and now a lot more work has shifted to Astra/Sol and I'm learning to make my workflows more resistant to the whims of each model.
I am using Opus 5.5 high almost exclusively. I definitely felt this and looked for nerfing watch sites. It felt like downgrading to the old version of deepseek. Things have improved a bit now.
The reverse happen with Deepseek. Getting smarter and faster especially if you dont mind the China API. Gemini Argon getting dumber compare to older 3.8 and significantly slower too.
The way Anthropic continues to get away with testing these nerfing "experiments" on customers without their consent depriving them of the model quality they've paid for is wrong.
The Windows Insider program has shown plenty of people will willingly sign up to be testers for a taste of the rawhide. Anthropic could do this in a transparent way but they continue to choose not too.
They can get away with it because they are so far ahead of OpenAI models for coding, it's not like you can switch to anything else if you don't like it. Astra is even worse than nerfed 5.5.
Pretty much the only alternative is loading up some credits and using Fable.
Astra when it hit the ground felt like “A Fable monent”: it was so good and concise and just was done with whatever task in a short time, where opus and fable sometimes can take forever. I think it lasted a week or two until it started to deteriorate to where we are now. Coincided nicely with the stopping of codex20 subscriptions etc.
Sol/Astra are sweet but they also have pretty much no discipline or executive function, which is where Opus 5.5 comes in, and I've not seen that capability in it nerfed too bad in my experience.
I have high hopes for Carbon here -- Argon is not quite Opus 5.5 but it's pretty darn good.
Every time I read something like this that doesn't match my experience I'm bewildered until I remember that providers play quality games and that everybody's workload and harness is different. For me GPT 6.1 Sol is tremendous value and excellent at low-level work like DSP and 3D game engines whereas Opus is more expensive and not as capable. I also don't get along with DeepSeek at all even though everybody on HN seems to think it's the second coming.
A neutral third party who runs a matrix of closed-source coding benchmarks across multiple harnesses would be helpful but that'd take a lot of time and money...
My experience is that 6.1 Sol is a little behind Opus 5.5 but it’s in spitting range. Their cost per task is generally higher, but if you’re on a quota plan the difference in cost doesn’t really matter until you’re paying API rates.
Both get the job done these days. I still hate Opus 5.5’s output, 6.1 Sol does an acceptable job of doing a second cleanup pass to it.
How can you even tell? Do you have an experiment I could run right now which would would prove it to myself.
Yes this is the experiment I have been running for a while, that shows the constant comments on Claude models here have to be push from Claude team friends...The experience of output of OpenAI models is consistently better.
1)- Get a coding task and get the output of Opus and Astra.
2)- Open a new session and ask each other to review in detail the solution from the other model without mentioning where is it coming from.
3)- After the review, ask the original model to comment on it and state which solution is better and if the criticisms are valid...all in a new session.
4)- At the end Opus and others from Anthropic always end up agreeing the OpenAI model solution is better. My human review agrees.
After doing this for a few weeks...and consistently having their models rating higher the OpenAI solutions...I cancelled the Anthropic subscription...
Isn't this inherently task/domain/programming language/even prompting style specific? Why can't other people have genuinely different experience than you?
Its was not about my experience is about my experiment
Nice try Amodei, but buying openai or anthropic credits is equivalent to just setting your money on fire.
DeepSeek, Qwen.
Yep, just one hour ago I again reached my weekly limit on claude code (3 days till reset btw), using Fable as Head-Orchestrator + Codex sub-agents via custom mcp.
I tried $20 of usage credits as a test once the limit was reached.
I shit you not, it lasted for less than 5 minutes.
Yeah it's a compete joke/scam. I bought $200 of credits once... gone in an hour. Never again. Part of me is thinking they decreased usage partly to force users to buy credits before the IPO... but at least in this case they ended up losing a user.
I get 5x more done (of equivalent quality) on my opencode go + DeepSeek flash sub for $40, than $200 anthropic sub.
Have you had a chance to try DeepSeek as a Orchestration/Manager for sub agents?
I have been using DeepSeek flash for a few projects and it is actually crazy capable, And I don't think I have spent $20 via api in a month yet.
Yeah it works great, maybe not quite as good as opus 5, but it's literally 20x-50x cheaper, and several times faster, an extra review/refinement loop per task brings the work up to opus level while still being much faster and cheaper overall. I have my own setup/tool (which I hope to open source soon) for planning/tasking/tracking which I found helpful to keep things on track an not forget/redo things - regardless of model - essentially tasks, dependencies/hierarchy, priorities, requirements, and task completion evidence tracked in the repo in structured plaintext/markdown files. DeepSeek was able to very easily understand and follow the workflow right off the bat.
Ok, thanks, I'm going to try to setup the pi harness and add on some needed extensions for my workflow. I'd appreaciate any advice in regards to any addons you would recommend if you have them.
[dead]
They'd probably claim "well, it's not an A/B test then, because we are pre selecting for a specific user type who signs up for this".
I don't think that's an excuse, but it is likely the reasoning.
Probably won't work for Anthropic because they likely want a larger assortment of users and use cases.
Then they should suffer the consequences of not delivering what users paid for.
If you aren't pay per token, then they can do whatever they want.
A bit like "You can order the steak for $75, or you can have whatever the chef is making for $30"
Eh sometimes it cuts the other way, like when Opus 5.5 was being shadow-routed on Opus 5. I am on a team plan and seem to not get any of these A/B tests.
Oh brother, do I have news for you...
[dead]
You would prefer they experiment on everyone at the same time without a control group? Think about your answer.
No, they should not change the model at all when you select the model.
No improvements to prompts, tooling, harness between model releases? Is this really what you want? I don't think you've thought this through at all.
Please make your substantive points without crossing into swipes or putdowns.
https://news.ycombinator.com/newsguidelines.html
I don’t think I said anything about prompts, tooling, or harnesses.
The model itself should not be substituted(for example, quantized) without an explicit SKU change.
It isn't though.
They should let you use your subscription with your own harness. Until they do, you're just borrowing someone else's workflow, which is liable and likely to change whenever the fellas over at Anthropic get a wild hair and decide to introduce a new experiment.
Well.. that's a totally different argument.
I agree, I also noticed visible drop in quality, a week before I was able to ship two apps and when I was working on the third (which is almost as much big as the first) it started acting weird, I even started a new session thinking it was because of hallucination... It did give me the output but burnt through my tokens earlier than I expected. It was even missing and getting confused between branches... All in all doing more mistakes... One point I would like to bring to your attention is, it has started using background agents a lot lately (maybe because I prompted it to use Sonnet models for scripting and Opus will supervise, but still it was better in the earliest week.)
We use Azure Foundry hosted Opus and it seems as capable as ever.
I run a small local GPU cluster for Qwen 3.8 Flash and while it's "free" (you should check electricity prices in EU...) unfortunately it doesn't even compare to Opus 5.5 or Astra in capabilities. It's like GPT 6 vs GPT ~4, but imagine GPT 4 is also 100x slower and sounds like an airplane taking off.
In the end it's totally possible that I would pay more for electricity running Qwen for 24 hours churning on some complex problem vs. Astra one-shotting it in 15 minutes.
I hope we get there, but it's not there yet, and I'm sure never will be with consumer GPUs. Maybe if you spend $20k+ on a local AI rig. There's just no market force to give access to AI to everyone at fair price.
Ever since I started using Claude at its inception this was the biggest breach of trust that I've experienced.
I'm afraid to have touch my code base because a nerfed model could slip in stupid bugs.
This is also why a weaker local model might be preferable because it allows me to calibrate trust, and my expectations don't change.
I have had Astra analyze my long running Opus conversations and the degradation definitely happened on November 6th and 7th.
From what I understand they are picking and choosing who is running the latest models. So nerfed yes but for you not for everyone
Sadly this thing stopped running a few weeks ago, the propduct half life is truly low these days otherwise wed have some data to rely on: https://marginlab.ai/trackers/claude-code/
This is not real model's reasoning chain, they might have just replaced or tuned the model that summarizes it. If you want to track actual degradation you need to run evals. Trackers like this do exist, e.g. https://aistupidlevel.info/
Anecdotal, but I tend to get by with medium thinking on Opus and I noticed that as of sometime last week it really did feel like it had become noticeably slower to think about things.
I wouldn't say I've especially noticed its effectiveness having been reduced, it's just slower.
We are on the cusp of an intellectual obesity epidemic.
We need a consistent way to tell level of service for LLM services. Without they can play with us and steal money by providing different/quantized models than we paid for. I believe this may be a good next AI company if they will be reliable.
They're preparing to release a new model, so it has to look better than the current one.
Yes, this take has consistently held true. Indicator to hold out major greenfields and just wait it out.
I've noticed I've been running into more safety guard rails the last few days. I've been using fable 5.1 for scientific lit search (e.g., on topic X, give me a table with links to the top 20 papers, and then major criticisms and rebuttals with direct links to the papers, etc.) to just save me time in getting acquainted with a new sub-topic. The last couple of days it's not just been dropping me from 5.1 to opus 5.5, but all the way to opus 5, stating that it's for safety reasons.
I have noticed a mass lobotomy of Anthropic models recently. The frustrating part is that the target keeps changing and however you work with it today, it can fail tomorrow.
Hm, I’ve seen a degradation of 5.5 yesterday. I’ve had a session running for around 50 days and it just started to generate garbage and ignored all rules. A fresh one feels the same as always
not as bad as sol 6.1. im getting better results out of glm 5.3
I was down voted into oblivion when I said that 3 weeks ago. Both Codex and Claude are absolutely nothing compared to what it was.
How can anyone build on this quicksand where you never know whether the same model will still be available tomorrow? The whole industry has become a casino.
What I'm surprised by is that no-one has mentioned that FAANG companies run experiments (often multiple overlapping ones) on 99% of the traffic. You're almost never getting "vanilla". And Anthropic is full of talent from those places.
If you look at any sort of tracker, there's no telling whether you're in the same experiment group as the tracker (most likely not).
this is the reason i cancelled claude after 4.6, non-determinism aside, anthropic just doesn't seem care about maintaining sane baseline for their products
This could explain needing to blow through my limits last week. So much of it’s outputs on xhigh needed serious revision, and took much longer than usual to meet specs.
I’m only prototyping so most of the code will be discarded, but even in this circumstances you can’t allow too much sprawl and decoherence within a single project. It makes prototyping cost so much more over time. It was pretty frustrating.
I noticed a massive quality change a week or so ago. Major downgrade
Exactly the reason I had previously cancelled my openai subscription
I'm looking into Gemini
Sometimes I feel like the model is not necessarily needed, but the harness might be. Asking questions/permissions more often, delegating less, creating fewer sub agents. Just overall making workflows slower/harder to review. The main model might be just as smart/fast, bur it’s like the agentic side that starts crapping out
https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
If Anthropic indeed nerfed Opus 5.5 on November 3, it could be because they wanted to be prepared for inference cost going up due to the data center moratoriums and the election results. I really hope this is false, but I don't see how inference cost can go down in the near future.
1) it was certain for a long time before the election that the unwholesome party is going to get slaughtered
2) the wholesome party is not in power yet
3) even when they are, the orange man will veto everything they might attempt to do
4) even when the orange man is replaced with someone wholesome, and that wholesome person enacts a bunch of wholesome legislations, data centers will simply move literally anywhere else on Earth.
Regarding 4, I don't think anyone in the US, regardless of how they voted, would want a bunch of SOTA GPUs being in any other country than the US. It's either they are all here in the US, or they are not anywhere. As a result, if data centers cannot be built in the US, the GPUs will not be produced only to be exported elsewhere.
Several datacenters were stopped directly due to the elections: https://www.theguardian.com/us-news/2026/nov/04/datacenters-...
stopped before they were even built, yes.
>In Ohio alone, at least 14 municipalities passed bans or restrictions on datacenter development, including in Sunbury, where Amazon has proposed a $2bn datacenter project. In Maine, eight municipalities passed temporary datacenter moratoriums. In Emporia, Kansas, where a 1,000-acre gigawatt-scale datacenter was recently proposed, a ban on high-impact datacenters was approved by voters.
and now they will be built elsewhere.
I think there's a rather obvious precedent for that inevitable outcome.
I mean, sure, they'll be built somewhere else (and I think the approach of "stopping" rather than "protecting residents from price changes" is terrible), but buying land, permitting, etc take years and tons of money. It's going to slow down the industry a bit.
It has been nerfed for a long time now. Anyone who tried original Fable 5, knows how good that was, and all subsequent models, including current Fable and Opus have definitely been nerfed. It's like they add a mandatory 5% error added to all code, etc...
It's very frustrating, even at max effort. It used to do things correctly on its own, now I need to keep reminding it to do obvious things like "Make sure you are not dropping steps as you rewrite the docs". It became unreliable, like how Opus 5 was.
A official statement is much required
On November 4th, Claude decided to leak all val data into training data, re-train, and overwrite the output models which weren't in git.
My mistake for not keeping it in git, but it nuked 2 weeks of work.
You have to keep the model unchanged from when you released it. Changes have to be mandated to roll over to a new version. Otherwise all of this is useless, you have to atleast be “deterministic” with the model capabilities.
Ok, I'm not going crazy then. Noticed that it is slower and results were of a lower quality.
There's this ongoing theory that Anthropic makes it's model dumber on purpose to make the difference noticeable with new model release. It has been documented by someone, you could see performance being dropped
Its coming a time that these commercial offerings will have to provide independent daily benchmarking to ensure their numbers align every day, with the risk of being called out for defrauding customers.
Somewhat related:
Livenerf: Has Opus 5.5 been nerfed yet? - https://news.ycombinator.com/item?id=49901736 - Sept 2026 (394 comments)
Maybe not? https://github.com/ninjahawk/livenerf
Nerf bench tracking model benchmark performance over time: https://www.bridgebench.ai/nerf-bench
Personally, I feel that Opus 5.5 is starting to behave like the old Opus 5. Lately, the model has been using a lot more of the compact, hard-to-read jargon that is typical in Opus 5
If this is real, it is straight up violation of implicit contract. When you pay for a service you expect stable and reliable performance. Intentional degradation of service is basically fraud.
I have a small eval set that I rerun every week with my own tasks. That's the only way I've found so far to distinguish between model changes and having a bad day.
I feel strongly that GPT-6 Luna has been nerfed. I was super-impressed by it in the beginning, now it seems to think overly much and get absolutely nowhere with even very simple tasks.
It's hard to tell, but it felt the same to me. Either way, Fable 5.2 just came out so abnormalities might have caused by infra-side changes related to that.
Feeling like it's better on my end :shrug:
On some long running conversations it seems to better understand the end goal, follow the constraints we have established.
Yes, it feels like that as earlier, the "low" effort was able to do the great work for which now I need to use "medium".
LLMs are such a great business, they are non-deterministic, hence you can do whatever you want, because customers can't do anything about it:
* nerf them based on supply and demand, when you have a high demand, nerf it, but keep charging same price.
* do you want to show revenue growth to investors? increase thinking tokens, customers will keep paying, because again, this is non-deterministic system, sometimes it thinks more
* do you want to increase profits? start serving 4-bit, 5-bit quantizations and keep charging same price
* want even more? publish a paper "Advanced Auto Super Smart Prompt Router - AASSPiRation", and start routing requests to Qwen3.8-27B or GemmaE2B from time to time on the edge
* Not enough? introduce "Complex Chained Workflows - CCW" in your coding harness, which spawns multiple deterministic workflows (which you can't stop or control) and they will use Haiku to write the code, then use Opus 5.5 high to verify the code (which is obviously wrong) and Opus 5.5 produces recommendations on how to fix the code, delegates to Haiku, which again fails writing proper code, and keep doing it 5-6 times, you made lots and lots of money (don't forget, all should run in parallel, to pick one best result among 10 outputs).
You can say the same thing about nearly any business. Lyft and Uber, for instance. You can lower quality, raise price, whatever. This is why open markets matter so much - when you have competition, businesses can't materially worsen their customers' outcomes for long or those customers switch.
You can't do this with Uber/Lyft.
How do you even lower the quality - serve half broken car for a luxury order? No way person who orders luxury car accepts it.
Or do you increase wait times? - sure, order gets cancelled, you won't get a payment for that.
There are of course ways, but you still can do something, in case of LLMs, you blame yourself for poor prompting and retry, which again makes money to LLM providers
I think you're arguing without considering here.
- You can lower the quality standard of which car counts as which class. There are many classes of vehicle on both platforms.
- You can delete or not consider low ratings from certain users. Both companies do this already.
- You can increase wait times, of course. It doesn't at all follow that the order gets cancelled. It depends on the market and what each company believes the other is doing.
That's thirty seconds of brainstorming.
Uber/Lyft are infamous for using their control over both ends of the transaction to milk both ends for money. They have heavy use of manipulative dark patterns. They know the entire market on both ends so they can manipulate the spread to optimize profit for themselves... and they can hide much of it simply by not telling users.
Eagerly awaiting Artificial Analysis to redo their Anthropic benchmarks and kick it down 15-20 index points, in the interests of accuracy. Those guys love accuracy. They moved Heaven and Earth to kick Kimi off the #2 spot, out of their deep, abiding passion for having the most accurate benchmarks in the industry.
I have no doubt they will ensure their users get an accurate picture of Opus 5.5 within hours.
If Anthropic are doing A/B testing it almost certainly only affects subscription users, not API users. Every benchmark I know of uses the API.
Which means the only numbers anyone can cite are for a product most of the people complaining aren't using.
IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.
The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.
I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's language from the EU "dual quality" food scandal, which saw food brands (Coca-Cola was one) adulterate their recipes to use cheaper ingredients in select markets—though the external packaging was kept the same.
https://ec.europa.eu/commission/presscorner/detail/en/qanda_...
(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").
If I were a cynic I’d also say that it wouldn’t be particularly difficult for them to detect when a benchmark/eval context is active and to route those requests to larger models or higher quants.
Model releases remind me of https://commons.wikimedia.org/wiki/File:Shepard_Tone_spectru...
This'll likely be the reality moving forward - they allocate more compute at launch to get good test scores, reviews, and news coverge, they get people excited and used to it, then they cut costs.
I'm reminded of the video game industry where they would launch a game without any in-game purchases at launch, get the reviews to make note of it and give it a higher rating because of it... then add in-game purchases anyway. I think that stopped again though.
I look at these threads as the other side of the "LLMs are super-intelligence" movement. If my superhuman champion has disappointed me in some way, it must be nefarious interference. The unpredictability of LLMs is immanent in the technology, but the movement elides this for its own purposes.
It’s A/B experiments at scale to see how many people put up with endless quantisation, a defence mechanism so they can say “hey all these wildly unjustifiable salaries for developers and management are totally unwild and justifiable, why look at how much we’ve slashed our inference costs”
I'm skeptical about nerfing. Are we saying they are instructing AWS to nerf the version of the model they're running on Bedrock?
If they did, there's no way that remains a secret.
If they didn't, Anthropic would be making their own service noticeably shitter than AWS Bedrock
Pretty nice business model tbh. Whenever you need/feel like it, just tune it down.
Oh, even psychological effect would be nice if you slightly tune down the current frontier 1-2 weeks before releasing the new one.
This tool I use to code/think/write for me doesn't work as well as it used to. Can I just go back to coding/thinking/writing myself?
You know what doesn’t get nerfed? DeepSeek. Costs me maybe 0.50 per day if I’m using it a lot.
I just don’t see the point of using these expensive models anymore.
I’m wary of this whole "watermarking" thing. I don’t believe it won’t end up having a negative impact on the quality of the responses.
Isn’t this just like when people think their IPhones start getting sluggish whenever a new IPhone model is about to be released?
It is like that yes.
https://www.businessinsider.com/apple-batterygate-500m-lawsu...
Same here. I've noticed that the output quality has changed about two weeks ago, in the latest days the thinking quality is super low.
They probably nerfed their internal compute usage to justify their data center spend to the new shareholders now that they're IPOing.
Note to future self: When nerfing is detected it might be resources being allocated towards the next impending model release.
Does nerfing usually come before a new model release?
> "One Bug. Two effort levels."
Sounds like analysis fully driven by a model, which can be fine, if you know what you are getting into and can verify the output. Just last week had someone ask Claude Opus 5.5 on High about watermarking after I informed him and he'd rather believe a models output over me, do a basic Google search or spend the effort reading Anthropics communications on the matter.
The model told him with no uncertainty that watermarking text was not implemented in any Anthropic models, whereupon he showed me and I then asked him to just ask the model to check again, ideally online. These models do still "hallucinate", they still have major gaps in their output not easily caught if you don't know the topic and interestingly, anything connected to LLMs, model providers, etc. tends to be where they do worst.
The annoying thing is, they still sound immensely authoritative across most any topic, but especially LLM stuff, which is such an easy way to get tricked. I have been for sure in the past and will be in the future, it's why I recommend having some incredibly niche hobby (like sewing with less known materials, obscure motorcycles, etc.) to check how quickly the model makes a mistake which no one in the know would make. Just to keep yourself disillusioned and inoculated.
In this case, looks like the LLM did not consider that these are (and have been for months now) just summaries and the actual reasoning can not be analyzed for encryption reasons.
If in doubt, please do some research independent of the model output. LLMs not having any internal "understanding" that'd give them anything akin to an internalized "feel" for what they "know" accurately enough is one of their major flaws.
Funnily enough, once did the opposite, made a manual analysis of a Github outage, then had the model just write an artifact to quickly share. Was lazy, could have just put something on Pages, but taking summary changes and extrapolating that to a "nerf" is less ideal.
On the regression front, I run my own, private evals and have only once encountered consistently reproducible, over run-to-run deviation, degraded performance, which was purely present when using Claude Opus 4.8 in April after some system prompt changes in Claude Code specifically which were addressed soon thereafter. These were not present with OpenCode via API and any other time the public on Reddit and co. reported regressions, I could not reproduce it, even when using Claude Code as the harness and a subscription.
What I have encountered and been able to reliably reproduce is that certain labs do very clear teach to the public benchmarks and underperform them commonly, whereas others do not. And that certain, especially cheaper and higher throughput, neoclouds very clearly take some quality affecting, tool call misformatting shortcuts.
For what it's worth, currently measuring, but haven't seen anything so far with Opus 5.5, especially nothing connected to reasoning that'd make the assessments some of the commenters draw from this apparent.
Yes, the activity on the CC github repo is mostly agents replying to each other. The signal/noise ratio is 1% at most.
I'm also noticing the model seems to be a lot more sloppy now and less reliable like compared to a week ago or so?
It's dumbing down a little bit but I don't think its a lot worse. I think the change in display adds to the feel
as a user I believe it: they are probably reducing assigned compute to prepare to offer new models.
That would be the less bad reason.
Usage limits on Max plan also feels like its reduced. I now approach my usage limits much faster.
People who never tried outsourcing before are discovering that there are risks associated with outsourcing.
I suppose they can achieve a benchmark then nerf the model to save compute once they've got the headline.
I assume that arbitrary rerouting won't occur with hosted inference like AWS Bedrock?
Has anyone used their new tool FlashProve to verify the weights haven't changed?
Fable 5.1 quality took a significant dive for me last week. I'm not imagining it.
They always nerf before they drop a new model to make it look better imho
Yes, I noticed it too, dumbing down continues, there goes their IPO.
Yeah I realized some decrease performance on htmls it generates
The quantization will continue until version number increases.
I guess it’s AB testing and we are not on the bad side (yet).
Welcome to the world of post-IPO AI. Now the 20% growth needs to come somewhere, and profitability would be a nice starter.
5.5 became useless last week, so frustrating
Nerf means something else in deep learning parlance.
Wasn’t there a platform that kept testing that ?
LiveNerf suggests not.
https://github.com/ninjahawk/livenerf
My paranoia tells me they A/B test users which means consistent benchmarks go through the high performance pipeline while regular requests fall back to the quantized / experimental / low cost version. I can see the main reason being the oldest one in the book, which is making more money.
It could also be a regional effect where systems are under higher load and hence dynamically routing to / provisioning lower precision instances; or some other factor where one is affected whereas someone else isn't.
It's not paranoia and would align with https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
I wonder as well whether would this pop up on nerfbench
Do you ever a/b test them? Run the same setup and prompt multiple times to see the output?
I noticed. Seems to be worse now.
Isn't this just the tried-and-true planned obsolescence playbook? The frontier labs need to capture as much value as they can from the next "hit", and that includes eliminating competition where they can. Including when it's their own model. I have zero evidence for this, but it seems patently obvious: Nerf Claude to drive usage to their next model, where they compound value and (theoretically) can serve more efficiently with each generation.
Yeah because fable 5.2 is here
Anecdotally, it's currently feeling nerfed. The model keeps restating my intent as if it had thought of it. Output accuracy is slightly down, but the increased friction in communication is having an impact on my worfklow. Very annoying. Also, it flat out ignored system claude.md several times over the weekend. These observations hold between my personal pro and works enterprise.
I am curious. How do people figure this out?
I use Sonnet 5.5 daily and Opus rarely. My usage is also a little involved as in I do all the design and strategy and get Sonnet to just write the code
For that, Sonnet has been consistent
I use Opus or Fable when it's a dumpster fire situation where we have no time to think even
So far I've never had an issue with Sonnet even.
Namber leelavathi
These kinds of threads always make me feel like I'm watching some satire. Pure consumerism, like the people that used to fight in line on Black Friday. Maybe Anthropic is nerfing the models, why are you so addicted to them that you care?
Anecdotes and conspiracies and mass hysteria abound imo. And probably a bit of astroturfing from competitors. Models are not deterministic, nor are your projects. I work on code constantly for years - and across every anthropic and openai model people claim the same thing - yet I've never experienced this degredation, not even one hint of it. I would be acutely aware of it.
Variable token usage? Sure. To be expected in non-deterministic systems like these.
Distinct increase in errors and issues? Never. Not within a single model's lifespan have I ever experienced this.
And this whole notion of the A/B testing of performance/usage is completely ridiculous. There is zero evidence it has anything to do with performance. There are a trillion things to A/B test besides things that hamstring groups of users, so this is just being used to confirm the bias of people already caught up in these "theories."
I await any realistic indication.
Perhaps most importantly, human bias is not deterministic either. Everything seems consistent with people getting excited by the hype (or just the novelty) when a new model is released, and then coming back to reality a little while later.
If they're messing with models like this, surely they'd at least occasionally make a tweak that improves intelligence rather than degrading it. But I never see anyone say Model 4.2 got smarter a week ago or whatever.
lol very good point
It's gone pretty shit, I made a comment about this yesterday: https://news.ycombinator.com/item?id=50412897
Instead of being Opus on high (effort), it's Opus is high.
I also noticed a change. Normally it said, "Found XYZ. Now I do ABC". Now it says "Found XYZ. Now I will do ABC." I liked the robotic "Now I do ABC" better.
People are already so used to have genies and unicorns around they now feel infuriated when they are a bit slower than before.
Dude, you have a mythical creature at your disposal at retail, chill, it's going to be ok.
i’m so tired of this
all the measurements always say it isn’t
anyone who grew up playing dungeons and dragons gets that this is just people saying “you don’t understand, these five used to roll high”
every llm space is inundated with rank amateurs explaining at great length how it used to be better yesterday
nobody has the courage to clean out the obvious bullshit, so the drone gets ever louder and the signal gets ever rarer
do you remember what programming was two years ago?
stop complaining so much
Seems fine to me
Can we get a self-hostable distilled opus 5.5 yet please?
This company does not deserve to be successful and has no moral high-ground.
Say they did nerf it, why would they tell you? Why would they tell anyone?
Or maybe they didn't nerf it, and it's the same as yesterday, but you had a spate of bad sessions due to the whims of probability and statistics - why would they bother telling you that nothing has changed and the problem is you're unlucky? Do you think they want to remind you your experience of their product might be crap and no one can do anything about it? Do you think you'll take kindly to that? Do you think in the large, that a large number of people who aren't you would largely take kindly to receiving such news? Do they want that support burden? (They don't.)
Are we understanding statistics yet? You're only one person in a sea of potentially affected or non-affected people who are each potentially affected to different degrees or not at all, and it's a probabilistic machine they're selling which is used enough that, due solely to probability, some users will roll snake eyes for days.
Say that actually all this was wrong and actually, they ARE A/B testing and that caused SOME people to get unintentionally nerfed as a side-effect of one test (how many tests are underway at once by the way?). Why would they ever admit to that? "We noticed you got non-consentually subjected to our product-improvement bullshit. Nothing you can do! It's in the EULA!"
Besides which how would they even know who was affected? If they did, would they do a targeted apology, or apologise to everyone just in case?
(There's precedent here: when the thinking level default was lowered in Claude Code, there WAS a noticeable and broad effect, and there was a (admittedly quiet) apology and a fix issued. There was also a fix issued when the CC prompt got insanely wordy - the "terse" tone of voice mode, a new flag you had to update to get, and set manually.)
Let's bring in some game theory.
Let's put ourselves in an alternative hypothetical universe where everyone is going for the "trust angle" and thus publishes numbers proving non-nerfing. Initially every provider spends time, money, and energy - but the industry is so competitive so every lab needs every bum in every seat working on the product, so eventually SOMEONE stops and then there's no incentive for anyone else to keep doing it. Know what happens? The whole thing collapses. Being public with non-nerfing claims precludes A/B testing, putting whoever can't A/B test at a disadvantage. And also, when the numbers get withdrawn after people expected to see them it's a super bad look. So why would anyone bother in the first place? We're back to our own universe.
Silence is signal.
Do yourself a favor and ignore rigor-free superstition and anecdotally driven frustration and kvetching from folks with NO skin in the game.
ESPECIALLY anyone who misunderstands probability while at the helm of a probabilistic machine.
The more you listen to people tell you stories based on one single experience without any basis in statistics, the more you'll believe things rooted in opinion over things rooted in truth. That way lies madness. Steer clear!
[flagged]
I thinks more likely ya'll are experiencing a phenomenon that has been discussed on here before.
At the model release people become high/manic/addicted to the newest model think they now have some new superpower and then that "high" wears off and you think the model isnt as impressive.
The truth is llms have a slot machine-esque UX to them, and the labs designed it this way.
Dont worry you'll get your new model with some mystical name any day now and a month or two afterwards you'll be posting about how it no longer feels as good to use.
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[dead]
[dead]
[flagged]
Each new model version hits your dopamine receptors very hard when it first comes out, but in the nature of things it stops hitting after a while and you need novel models to fire them off again. That's why people think the models are getting nerfed after a couple of weeks; it's actually their endocrine system getting nerfed.
There were no noticeable complaints about Opus 5.5 before the end of last week. Large volumes of people are now reporting the same thing at the same time. If your theory is true then multitudes of endocrine systems separated by vast distances and vastly different initial Anthropic exposure times are perfectly in sync. Resistance is futile, you will be assimilated.
One of the best scenes in The Wire was when Avon explained to D'Angelo that the new shipment of heroin was just the old shipment of heroin cut with different baby powder.
Is there a practical reason this link uses twitter.com rather than x.com, or is it a nod to the old name? If it’s the latter, I don’t really see the point.
5.5 is useless below max effort, just frustrating and plain old wrong
[flagged]
IPO is coming so they will have to maximize profits going forward which will mean constraining the compute they give everyone to ensure the subscriptions make money.
I don't think the subscriptions will ever make money.
The subscriptions are a farm for generating training data.
I think the subscriptions are already making money.
Based on API pricing per tokens, I'm getting anywhere from $10-20k/month in surplus value from a single $200 subscription.
How are they making money in your view?
By selling subscription tokens for less than they cost to generate, of course.
What you said doesn't yield any information whatsoever about whether or not subscriptions are profitable, all that says is that subscriptions are less profitable than API.
Things are able to be less profitable than other things while still being profitable.
If something like the API is very very profitable, it's even possible for profitable things to be much less profitable!
Inference costs are known at this point. Even if you don't know their exact cost you can reasonably estimate the upper and lower bounds of that range.
Not sure what you're on about.
Right, and subscription pricing fits comfortably in that range.
Not sure what you were on about with API pricing.
I didn't realize I confused you.
Tokens are tokens, whether they're used on a subscription or via API.
When you purchase a subscription, you get a certain amount of usage. You're not told what that usage level is.
You do know how many tokens you use.
So when you calculate how many tokens you've used on a fixed-rate subscription, and use API costs to estimate how much it would have cost you without a subscription, you get what I referred to as the API cost of the token, or what people are actually paying for which would seem to give an indication as to the "market price". That means the price the market will pay for it.
I hope that clears it up.
None of what you said has anything to do with whether or not subscriptions make money. API pricing doesn't enter into it.
If I sell you a russet potato for a million dollars, that doesn't mean supermarkets selling the same potato for $3 aren't also making money.
Diving deeper into your non sequiter does not clear up confusion.
But are they profitable?
That's generally what "making money" means. Profit, not revenue.