> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.
Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".
It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B.
Here we re-run on precisely the same set of Terminal Bench 2.1 problems.
It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution.
But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.
Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates.
That said, I am open to new ideas.
Yah, prediction interval instead, right? (I'm still learning statistics)
Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.
It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).
So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.
I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.
If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound.
The kv cache was still bf16, so a few ops are the same between quants.
Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.
I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!
Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.
Runs the 3bit model faster than the 2bit one runs on my old-ass card. Can’t vouch for its intelligence yet, but i suspect whatever loss in smarts it takes is made up for by the extra resolution.
I would be very interested in a similar benchmark for *KV cache* quantizations.
I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.
I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.
On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.
I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams.
Also have about 2Gb vram left for use of OS.
my coding agents regularly reach 200k context used without noticeable degradation.
hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?
Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),
But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?
Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?
I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.
So, if there are irrelevant remarks, these are mine. :)
Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).
IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.
The line is "Is this an interesting and accurate article that concisely makes it's case".
LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.
I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.
(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)
-----
Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.
My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem.
In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.
Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.
I don't think so, no, because it extends to LLM-generated text I want to recall.
I use LLMs to generate starter/tutorial material. I may hate the way Claude writes but I absolutely don't hate the way Gemma 4 writes. But I have to continuously consult it in a way that I do not with human-written text, which gets its message across in a more persistent way I find less troubling.
(This non-memorability extends to AI images and video.)
You are right that there are some confounding factors in my life but while I was worried about middle age affecting recall, I find actually I am still remembering stuff humans wrote pretty much fine.
ETA: there is one thing that I have noticed that does affect recall that is specific to LLMs: watching text roll out word-by-word in LLM chat, I think, damages recall. It's engaging the wrong kind of memory and focus.
So I now let it generate and I'm trying to find decent ways to format it e.g. as PDF, to give it the best chance.
This could be generational; there were many studies twenty years ago that suggest that people a decade or so younger than me who grew up with full colour books and magazines and multimedia can read less-linear text layouts more comfortably, for example, so it's not out of the question that there's something generational going on here too.
Sure, but if you're going to publish it, at least run it through an edit prompt and tell it to remove clickbait "Its not X, its Y" rubbish. Like literally calling examples clickbait in the prompt has given me better results. Interestingly, I have a lot less trouble with the first draft with Qwen than with Opus.
Yes, LLMs make it a lot easier to produce a lot more garbage. That's not an exception to my point. If something is well written then it doesn't waste the reader's time.
Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?
Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.
I normally don't appreciate snarky comments on HN, me being but a simple curmudgeon wanting pure information and the occasional opinion. but this one actually makes a good point, even if it's not stated explicitly. Reading this book is a wonderful experience because of the way it's written. Reading the plot summary (from an AI or otherwise) yields nothing but the dry husk of that experience.
It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.
General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.
What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?
Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?
I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution.
To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.
The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.
Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.
But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.
Depends on the actual audience, I guess. My stated “wisdom” comes in part from
/r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.
But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.
> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.
Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?
Those are multiple-choice questions. If some of them are "trick questions", where obvious answer (e.g. the value taken directly from question's text) is wrong, bad model might perform worse than a dice.
On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.
If you want to do a 1-bit model you have to QAT at pre-training with way more data than chinchilla to compensate for the cliffs (like 50x). Quantization on an existing pre-trained model will almost always collapse at 1-bit
I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.
I’m more curious how each 4 bit quant compares. It seems like NVFP4 outperforms Q4_K_M in terms of speed and top 1 but is only good for expensive Nvidia cards
Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.
It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.
Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
I've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.
Thanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.
I leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf
I've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster
Too late to edit, but a few other things to note: I minmaxed the draft config. On my typical coding workloads it gets around 70% acceptance, more variable on prose.
The chat template is froggeric's fixed qwen template, v22.5 as of today.
Beats Opus 4.6 at what exactly? It certainly isn't code.
I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.
> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.
Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).
Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".
It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.
Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.
That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).
Yah, prediction interval instead, right? (I'm still learning statistics)
Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
Thank you for these, coz I learned a lot! Great that they are open access.
Mind blown. The more I read about statistics, the less I know.
“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)
the 2nd derivate of shannon covariance of noise begs to differ
This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.
It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).
So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.
I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.
Thanks for doing and sharing the research!
As they say, time is money.
In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!
Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.
If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound. The kv cache was still bf16, so a few ops are the same between quants.
Computer science has known the tradeoffs between memory and compute since ages ago. The same could be reflected here.
Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.
I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.
I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
Fair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.
If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.
There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.
Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!
What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates
Running Q3 on 5060ti with 64k context. It runs great
There is an interesting new dynamic 3 bit quantisation I have been meaning to test:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.
Q3 XL and Q3 XS are the two I'm trying to decide on
You might want to test this new dynamic GGUF:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
(I don’t know much about it, just saw a YouTube video about it last night)
Another one to try:
https://huggingface.co/Jackrong/Qwopus3.8-27B-Flash-GGUF
Runs the 3bit model faster than the 2bit one runs on my old-ass card. Can’t vouch for its intelligence yet, but i suspect whatever loss in smarts it takes is made up for by the extra resolution.
I have a 5080, three OpenAI Pro token resets, and I’m on paternity leave. Astra seems pretty clever. Maybe I’ll give it a task.
I would be very interested in a similar benchmark for *KV cache* quantizations.
I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.
I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.
There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:
https://arxiv.org/html/2609.04098
On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.
I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.
my coding agents regularly reach 200k context used without noticeable degradation.
P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
You don't have to quantize all layers and all dimensions uniformly, FWIW
hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?
Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),
But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?
Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?
I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.
So, if there are irrelevant remarks, these are mine. :)
Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).
IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.
The line is "Is this an interesting and accurate article that concisely makes it's case".
LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.
I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.
(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)
-----
Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.
My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem.
In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.
Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.
I don't think so, no, because it extends to LLM-generated text I want to recall.
I use LLMs to generate starter/tutorial material. I may hate the way Claude writes but I absolutely don't hate the way Gemma 4 writes. But I have to continuously consult it in a way that I do not with human-written text, which gets its message across in a more persistent way I find less troubling.
(This non-memorability extends to AI images and video.)
You are right that there are some confounding factors in my life but while I was worried about middle age affecting recall, I find actually I am still remembering stuff humans wrote pretty much fine.
ETA: there is one thing that I have noticed that does affect recall that is specific to LLMs: watching text roll out word-by-word in LLM chat, I think, damages recall. It's engaging the wrong kind of memory and focus.
So I now let it generate and I'm trying to find decent ways to format it e.g. as PDF, to give it the best chance.
This could be generational; there were many studies twenty years ago that suggest that people a decade or so younger than me who grew up with full colour books and magazines and multimedia can read less-linear text layouts more comfortably, for example, so it's not out of the question that there's something generational going on here too.
Sure, but if you're going to publish it, at least run it through an edit prompt and tell it to remove clickbait "Its not X, its Y" rubbish. Like literally calling examples clickbait in the prompt has given me better results. Interestingly, I have a lot less trouble with the first draft with Qwen than with Opus.
Except that wasting the reader’s time became a lot easier with LLMs.
Yes, LLMs make it a lot easier to produce a lot more garbage. That's not an exception to my point. If something is well written then it doesn't waste the reader's time.
Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?
Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.
I normally don't appreciate snarky comments on HN, me being but a simple curmudgeon wanting pure information and the occasional opinion. but this one actually makes a good point, even if it's not stated explicitly. Reading this book is a wonderful experience because of the way it's written. Reading the plot summary (from an AI or otherwise) yields nothing but the dry husk of that experience.
Well, sadly it's off topic on HN where everyone seems to only read non fiction.
Or they don't admit to reading fiction...
I think the advice is the same regardless of AI use: read articles written by authors that have a history of high quality writing.
You don’t need anyone’s permission. You have only so much attention, why spend it wading through slop?
It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.
General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.
What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?
Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?
I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution.
To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.
The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.
Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.
But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.
I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.
Depends on the actual audience, I guess. My stated “wisdom” comes in part from /r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.
But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.
> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.
Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?
Those are multiple-choice questions. If some of them are "trick questions", where obvious answer (e.g. the value taken directly from question's text) is wrong, bad model might perform worse than a dice.
On the other hand, not sure where from 25% baseline for random answers come from. Since this is multiple-choice-out-of-4 test, random guessing should be correct in 1 in 15 cases, not 1 in 4.
If you want to do a 1-bit model you have to QAT at pre-training with way more data than chinchilla to compensate for the cliffs (like 50x). Quantization on an existing pre-trained model will almost always collapse at 1-bit
I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.
I’m more curious how each 4 bit quant compares. It seems like NVFP4 outperforms Q4_K_M in terms of speed and top 1 but is only good for expensive Nvidia cards
Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.
It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.
Could this be used to estimate how many fingers LLM have?
Those benchmarks are very interesting.
But is there any model that actually works in a decent way at quantization of 1?
Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.
> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
Even at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model, in my testing.
I've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.
Nice, which model quantization is this? Is it on huggingface?
MTPLX is this software: https://github.com/youssofal/MTPLX
I tried it with the author’s 4-bit quant of Qwen 3.8 27B: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... (but no need to download it manually; MTPLX will ask which one you want).
Thanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.
I leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf
Have you tried https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit ? PrismML is the only people i am aware of doing 1bit that is decent.
I've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster
I treat it like image gen. Send a prompt then come back in 40 minutes.
the 4-bit matching bf16 on terminal-bench is a useful data
I've been running Unsloth IQ3_S on my 5060ti with mmproj offloaded, getting 600-1000 prefill and 30-50 tg with this config:
Too late to edit, but a few other things to note: I minmaxed the draft config. On my typical coding workloads it gets around 70% acceptance, more variable on prose.
The chat template is froggeric's fixed qwen template, v22.5 as of today.
local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
Beats Opus 4.6 at what exactly? It certainly isn't code.
I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o...
Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...
Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.
If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.