Nonstop. There is too much wholesale reliance on LLMs to generate content. When I use LLMs for scientific writing, I approach it differently: I write the paragraph dirty, then ask an LLM to perform a minor rewrite for "clarity", using Claude's now retired Concise Mode. This has been a great approach for scientific writing. It tends to prevent these overly used turns of phrase, it makes sure that the writing is making the points I want to make, and shortens writing time by cleaning up the dirty edges of my grammar (especially since my writing can tend towards convoluted constructions). More artistic/creative writing, I'd probably not use it at all, because then, it's usually (for me) about rthym and emotional flow.
BTW It IS an effective rhetorical phrase, but given it's ubiquity in Claude's output, I have to avoid it.
Ahhh yeah, I’m not going to sit here and say I don’t use Claude to clean up my writing (ie make it actually coherently laid out). In all honestly I tend to write in a rambling stream of consciousness style across random scrap markdown files (ha), but point is taken.
I was thinking to myself as I was reading the post that here we go again - it's like how much can a man take. It's not that it's unclear... because it is easy to read. I also long for regular old human writing, warts and all.
I second this, I read too much AI slop already so when something triggers that part of my brain, at this stage I immediatley lose the capacity to engage outside of work, largly because it feels like work. Scrolling through, this article looks like it holds useful info. Info i'd likely love to engage with, but realistically I cannot force myself to spend my weekend reading more ai outout, even if human seeded.
Fair enough, nothing more infuriating than the sycophantic way LLMs write.
But I’d definitely be dishonest if I said I’d stop using LLMs to tidy my writing (out of principle or otherwise), my hold on the English language has seriously degenerated over the last eight years since I pivoted into IT from customer facing roles.
Impressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too.
One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.
Thanks, good call. I've switched both throughput charts to a log y-axis (they were already log on x), so the sublinear taper is actually readable now instead of getting flattened by the big prefill numbers up top.
On mlx-engine / mlx-lm: I'd wager it's not resolved upstream. The core bug here is a re-prefill on hybrid recurrent models, and it's not isolated to my setup. oMLX hit it, and llama.cpp has the same issue open right now (https://github.com/ggml-org/llama.cpp/issues/22746). When two independent engines trip on the same thing, it usually points at a shared architectural gap rather than a one-off, so I'd assume mlx-lm is worth checking too.
I'd rather have a ai-written article then no article at all. I think this is a real struggle. Writing an article takes time that could be spent on improving the product. Would I prefer a human article, yes. But if the alternative is "no article", then that's how it is.
I spent three weeks debugging why my Qwen 122B setup on an M3 Ultra was taking 3–5 minutes to generate the first token on follow-up messages (despite having a "warm" context).
The root cause wasn't the model, but three specific infrastructure bugs in my serving stack:
1. Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing a full re-compute every turn.
2. Interrupt Path: Streaming replies weren't persisted when the generation was interrupted, causing history divergence.
3. Checkpoint Poison: A background writer created unmatchable checkpoints that crowded out valid ones, triggering aggressive eviction.
After fixing these, prefill time dropped from minutes to sub-seconds (53k tokens cached, 33 tokens prefilled).
I've open-sourced the fork (qMLX) and a benchmarking tool to verify these numbers. Would love feedback on the hybrid attention caching strategy or any other edge cases I might have missed.
The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.
> a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
This is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.
They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.
I appreciate the amount of detail in the post, I think it's a useful addition to the space.
That said, I have to read LLM output all day all the time, and I would implore you to take the time to explore your own voice a bit more.
> Two separate things then happened, and it is worth keeping them apart.
Is one of those phrases claude spits out nonstop.
Nonstop. There is too much wholesale reliance on LLMs to generate content. When I use LLMs for scientific writing, I approach it differently: I write the paragraph dirty, then ask an LLM to perform a minor rewrite for "clarity", using Claude's now retired Concise Mode. This has been a great approach for scientific writing. It tends to prevent these overly used turns of phrase, it makes sure that the writing is making the points I want to make, and shortens writing time by cleaning up the dirty edges of my grammar (especially since my writing can tend towards convoluted constructions). More artistic/creative writing, I'd probably not use it at all, because then, it's usually (for me) about rthym and emotional flow.
BTW It IS an effective rhetorical phrase, but given it's ubiquity in Claude's output, I have to avoid it.
Ahhh yeah, I’m not going to sit here and say I don’t use Claude to clean up my writing (ie make it actually coherently laid out). In all honestly I tend to write in a rambling stream of consciousness style across random scrap markdown files (ha), but point is taken.
> In all honestly I tend to write in a rambling stream of consciousness style
At this point that'd be a welcome respite from every single blog post being written in the same exact AI tone.
I was thinking to myself as I was reading the post that here we go again - it's like how much can a man take. It's not that it's unclear... because it is easy to read. I also long for regular old human writing, warts and all.
I caught the title "The real work..." labelizing things in a sort of weird phrasing like this is the one I've seen a lot.
I second this, I read too much AI slop already so when something triggers that part of my brain, at this stage I immediatley lose the capacity to engage outside of work, largly because it feels like work. Scrolling through, this article looks like it holds useful info. Info i'd likely love to engage with, but realistically I cannot force myself to spend my weekend reading more ai outout, even if human seeded.
Fair enough, nothing more infuriating than the sycophantic way LLMs write.
But I’d definitely be dishonest if I said I’d stop using LLMs to tidy my writing (out of principle or otherwise), my hold on the English language has seriously degenerated over the last eight years since I pivoted into IT from customer facing roles.
It will continue to degenerate if you use LLMs to do the cleanup work.
I was triggered by the "Honest Numbers" section
Thats honestly a load bearing phrase, it truely wires clean sentences together, would be a footgun with an unknown blast radius if you didn't use it.
Let's find the seam.
[dead]
Impressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too.
One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.
Thanks, good call. I've switched both throughput charts to a log y-axis (they were already log on x), so the sublinear taper is actually readable now instead of getting flattened by the big prefill numbers up top.
On mlx-engine / mlx-lm: I'd wager it's not resolved upstream. The core bug here is a re-prefill on hybrid recurrent models, and it's not isolated to my setup. oMLX hit it, and llama.cpp has the same issue open right now (https://github.com/ggml-org/llama.cpp/issues/22746). When two independent engines trip on the same thing, it usually points at a shared architectural gap rather than a one-off, so I'd assume mlx-lm is worth checking too.
I'm not sure I'd call 1.5min processing compared to 3-5 min processing as "interactive".
Do you have any rough stats on how many total tokens per day you end up using with this setup?
Sorry but I'm not reading an AI slop article no matter how pertinent or interesting the subject is. You want my attention? Earn it. Write it yourself.
I'd rather have a ai-written article then no article at all. I think this is a real struggle. Writing an article takes time that could be spent on improving the product. Would I prefer a human article, yes. But if the alternative is "no article", then that's how it is.
It's fine, if you don't want to read it you don't have to read it, nobody's forcing you.
I spent three weeks debugging why my Qwen 122B setup on an M3 Ultra was taking 3–5 minutes to generate the first token on follow-up messages (despite having a "warm" context).
The root cause wasn't the model, but three specific infrastructure bugs in my serving stack:
1. Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing a full re-compute every turn.
2. Interrupt Path: Streaming replies weren't persisted when the generation was interrupted, causing history divergence.
3. Checkpoint Poison: A background writer created unmatchable checkpoints that crowded out valid ones, triggering aggressive eviction.
After fixing these, prefill time dropped from minutes to sub-seconds (53k tokens cached, 33 tokens prefilled).
I've open-sourced the fork (qMLX) and a benchmarking tool to verify these numbers. Would love feedback on the hybrid attention caching strategy or any other edge cases I might have missed.
The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.
> a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
This is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.
They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.
fascinating! That's a useful thing to know.
> The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache.
How is that counter-intuitive? You changed the key, so it didn't return a value. Thats a KV cache working as intended.
[dead]