I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
> Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
Speaking as somebody who is a performance geek, my knowledge came from relentless experimenting over the years (starting in the 8 bit era). Beyond the basics, I haven’t seen much on high-performance engineering online. To learn, you need to do the hard yards and I think performance tweaking becomes almost instinctive rather than something driven by a hard set of rules.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
I don't disagree, but IMO, a lot of code doesn't get to the point where those very low level techniques drive performance. Like, yes, if you are doing some heavy floating point math then that's where you end up needing it. However, in a lot of code finding hot paths and often simply switching out a O(n^2) for an O(n log n) or faster.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
I mean, sure, but it really does depend on what you're doing. If you're working on a library with collection-types and you want to make each iteration as fast as possible, then roll up your sleeves. If you're writing a compiler and you want your language's source-code to finish compiling this week, roll up the sleeves. If you're working on a game-engine and you want to draw more than everyone else, roll up the sleeves...
There are plenty of real-world reasons why you'd want to get knee deep in this stuff. I wasn't suggesting not using tools (I've literally spent the day buried in JetBrains' memory and tracing tools!), but those tools can only tell you what is happening now, not what to do to improve it.
Profiling is, of course, essential. But performance tweaking can be quite a laborious process: if you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course). Picking the correct data-structures is all in the 101 of performance engineering. That's in the literature. But it's all too basic and simplistic. Most performance minded engineers wouldn't need a profiling tool to know which data-structure to use.
At the smallest level there's a lot of mental theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
> but those tools can only tell you what is happening now, not what to do to improve it.
They tell you what's happening now, but they also tell you if what you've done has had a positive impact.
> If you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course).
I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems. Even in some of your examples like a compiler, game engine, or collection library, the big oh matters and if it's wrong, that can be a lot more important than shaving 0.1% on writing a function in a low level fashion. Big-Oh is gotten wrong a surprising amount of time even though it's 101 level stuff.
> At the smallest level there's a lot of theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
Oh because people get these things wrong all the time. That's why performance engineering stresses that you test, test, test and know what your testing and know why your testing could be wrong or corrupted. You should not trust your intuition because things change and it isn't always correct.
A good example of how easy it is to get measuring wrong. Imagine you start tweaking a function and you measure that your application became 1% faster. Was it the work you did on that function? Surprisingly, not always (at least not directly). Sometimes, it's the case that when you work on a function you re-align other functions as the machine code has to go it memory. It's possible that an undiscovered misaligned while loops was actually causing a large portion of your performance spill and by tweaking the function here, you aligned the while loop (or maybe a few of them). And, importantly, a new change somewhere else might re-unalign that same while loop.
You walk away thinking you've learn some low level lesson when in actuality your bit twiddling simply accidentally fixed something somewhere else.
This is why measuring is so important but also good measuring is even more important.
> I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems.
For what I work on, the hidden constant is often more important than big O. For example, a hash map has better complexity than just searching through a vector. But if the vector is small enough it will best the hash nap for actual time. Just plain searching until you find the element will even beat binary search on a sorted vector for small enough vectors. The reasons are complex, to do with cache, prefetch, branch prediction and also just how many instructions your tight inner loop has. (And the specific reasons will vary between desktop class CPUs and microcontrollers. But both exhibit this pattern.)
You could argue that at that point why bother optimising at all (there aren't a lot of elements in the collection after all). But there are two distinct cases I have come across over the years where it still matters (and for what I work with, they represent the common cases):
* You need to look up in a small collection a lot (either lots of lookups into a few small collections or a few lookups each into lots of different small collections, I have seen both cases).
* Hard realtime code where predictable latency matters. Hashmap has a bad worst case, binary trees and binary searching has badly predictable memory access patterns. And in this case the collections are usually small anyway (there are only so many actuators and sensors your equipment has, and/or the embedded microcontroller doesn't have a lot of memory anyway).
> But if the vector is small enough it will best the hash nap for actual time.
This will all depend on the size and type of object stored in a vector.
If you have a relatively small and flat struct that you are storing, then sure that will likely win. But if you are working with a collection of pointers, then the hash map will (almost) always win.
> Hard realtime code where predictable latency matters. Hashmap has a bad worst case
The hash map worst case is a linear search. It will be a lookup + the search. This also depends on the implementation. You could, for example, use a robinhood hash map which trades insertion times for lookup times.
A bad hashmap implementation will store collisions in linked lists. A better one will try and put them in a b-tree. An even better one will use a vector for collisions. And the most cache friendly version stores everything in the table and does probing on collisions.
There are specific cases where vectors are better, but those are the exception and not the rule in my experience.
It all depends. I was thinking mainly of small keys that are at most a machine word. But yes I conceded that for large keys (extremely unusual for what I do, thus I didn't think of that case before) a hashmap wins.
Also, while on a desktop class CPU any pointer chasing you do tends to dominate, that is not the case on small and medium microcontrollers (which I work with a lot). They have very short pipelines and their memory is all internal SRAM most of the time. Here classic cycle counting is still king, so you need to ask yourself if it is quicker to compare the key than to calculate the hash and compare hashes.
Code size in general matters a lot on embedded. Even a simple hashmap will always have more code than a simple vector. And a state of the art implementation with probing and tombstones (such as swisstable in C++ and hashbrown in Rust) will be way too large to be usable.
> Even in some of your examples like a compiler, game engine, or collection library, the big oh matters
Sure, but for example - today - I am literally building and optimising high-performance collections for my open-source library. Big-O is irrelevant, because I have built pretty much all of the fundamental collection types, what I care about is lower than that: what happens when enumerating any one of those collection types. Big-O tells you the scale of the problem, but not the per-element cost, which is still important when you're building core data-structures.
I am concerned about cache-friendly memory-layouts, how to do collection compositions without unnecessary memory allocations, keeping enough guards in place to make the types safe whilst removing as many branches as possible, reducing memory copying as much as possible, and catching stupid shit the compiler or JIT does and try to work around it. Literally what happens per-instruction, per-iteration, not how to pick a big-O based data-structure: trying to make all data-structures as fast as possible.
Anyway, we seem to be talking past each other. You're talking about the basics, I'm talking about the original source of this thread which was that (apparently) LLMs are surprisingly bad at optimisation. Which, I am trying to highlight becomes almost voodoo at a low-enough level and that highly-optimised code looks progressively more strange and opaque (in the hunt for a few nanoseconds here and there), which for an LLM wouldn't look statistically significant. I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.
> I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.
Surprisingly, it isn't. I catch the output of LLMs breaking these rules all the time. Just as it is pretty common in general programmer code.
The greatest sin I often find isn't necessarily Big-Oh related but rather multiple traversal problems. Much like regular programmers, LLMs love to do multiple passes over the same list to extract data. For example
let cats = pets.filter((p)->p.isCat());
let dogs = pets.filter((p)->p.isDog());
A lot of programmers are oblivious to that sort of performance issue. It comes up a lot.
I would think it's that the kinds of places which value this kind of knowledge often have major disincentive to share it. I'm thinking of HFT firms as one example.
Well if any labs want to scrape my knowledge of such things into model weights or faster training runs, I do consulting at movsoft.io :-)
It really is [as simple](https://github.com/ratatui/ratatui/pull/281#issuecomment-160...) as the Molly rocket video about terminals makes it seem; I am sure the models will catch up eventually. They have to overcome a lot of bad training data in most languages, but I find in Rust they usually do ok the first time.
I ran into the exact same wall when doing some SIMD + OpenMP optimisations.
Me: "Parallelise this loop without changing the results."
AI: "I can't do that, because of floating point accumulation."
Me: "So spin out a temporary fixed-length array, accumulate into that thread-locally, and then do an ordered summation of that at the end."
AI: "Oh, you're right!"
It's a fantastic tool for automating the implementation grind, but you still have to hand-feed it the strategy. I guess it will get there at some point.
A new meta is emerging of having high taste + infinite compute with little domain expertise. One recent example is Jarred Sumner taking a crack at the Riemann Hypothesis
> Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).
How much experience can you replace with infinite compute remains to be seen I guess.
I see the same thing, except I was working on high level performance reasoning. Whenever a piece of code has multiple steps that require multiple algorithms to work, AI almost always fails to guess which step is the slowest and what causes that step to be slow. Even Fable makes wrong guesses. You definitely need to give them a benchmark feedback loop.
Are there actually any humans who can reason about things like cache performance from first principles? I know there are some people who think they can, and I suspect they're fooling themselves. The one iron principle of micro-optimization at the level of cache hits is "measure, measure, measure", you just cannot think your way to the right answer on the first try. Processors and instruction sets today are too complicated, and tips that worked on one generation might be neutral or worse on the very next revision, making all the cargo cult knowledge passed around on this topic at best useless. I'll echo one of the sibling commenters here and say that LLMs probably bullshit their way to answers on questions like this because that's what humans online do as well.
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
Yes, people can demonstrably do this with high reliability.
Some humans carry detailed models of CPU microarchitectures in their heads, against which they can design code from first principles that will be nearly ideal on the first try. It is repeatable and verifiable. The best people can accurately predict the measured performance before writing a line of code.
Measurement is useful in cases where the model of software and hardware interaction is materially incomplete. In most cases this is because the people writing the software have insufficient understanding of the hardware. Having a limited understanding of the hardware is a choice.
I've been having great results with this kind of thing. I find that really just need a sensible framework within which the optimization can take place. Essentially just providing the measurement harness, and some sort of motivation for what I'm doing.
The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.
Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.
But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.
Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.
And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.
AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.
Agree it's amazing how much low-hanging performance fruit AI can trivially find. On the other hand though, once you get through the obvious no-brainer stuff, there's a lot of non-trivial tradeoffs in performance and I think that still demands a good amount of expertise to guide the AI in the right direction. Obviously AI will continue working it's way up the value chain, but I think there's a glass ceiling for AI where the right macro tradeoffs and perspectives on how software should work will bump into the hard and often articulated reality that different stakeholders want different things and often have either magical thinking or even self-deception about how those desires can co-exist with what everyone else wants.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
Yeah, the biggest example is performance optimizations that sacrifice your data model to the point that you'd never accept them.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
I think they can be useful for quickly iterating through benchmarks and trying lots of ideas, but they won't come up with them on their own. Also, I'm not sure why, maybe some mean reversion thing, but they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything. Once I point it at a tool, it can be ok at using it (I say ok because they seem to skim the help docs, which is truly ironic, considering I seem to read it more thoroughly even though I'm 100x slower at it. I assume this is some token saving system prompt), but they won't suggest it for you.
This is why I'm not worried about being replaced for now or the forseeable future. For all of the improvements they've made, this part just never seems to change. They could slap another heuristic prompt for the edge case, but eventually it'll revert to the mean again.
I think there is a way to use LLMs to help with programming, but not when I'm not the driver in the seat writing the tests and deciding the architecture. Also I would never ship code written by them as the final product for anything I care about. Since I, like most people, find reading code to be arduous. The more fun thing to do is to force yourself to rewrite it all, treating the LLM's work as a rough draft.
> they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything
Make sure you process doesn't depend on anyone reading your mind.
When I run into things like this, it becomes a one-liner in my instructions/harness or in the canned prompt/skill I use that sets off a process.
In this case, I instruct agents to proactively build/improve diagnostic tooling if it would help them with their task + if it meets a bar of generalization/reusability (else it should be an ephemeral probe that gets abandoned at the end of the solution).
> If it can be measured, then LLMs can optimize it
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
The point of this post is that this is explicitly not the case. If the metric is measured, the agent finds a way eventually (around 5 total tries typically unless it gets stuck), and learns from iterations where changes caused a regression after a revert.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
I was actually emphasizing a methodology like yours. There is a lot more going on in your post than "the metric is measured." And even then, there's no guarantee the model will do anything except burn all your tokens trying stuff, it just won't commit the failed attempts.
"Attempting" implies a high risk of failure. That is not the case with the workflow described in the article, and from other comments, the approach is generally reliable as well.
It's also not terrible on token usage for smaller projects.
I don't know why I have to say this again, but I am telling you that the workflow you underlined in the article is good. But the more you argue about it, the more it sounds like you want people to take a more generalized conclusion than what you actually demonstrated.
> "Attempting" implies a high risk of failure.
Of course it does? Your safeguards also imply a high risk of failure. You have restrictions that just rollback everything the LLM "attempts" to do.
That is not to say that the overall workflow is failure prone, but obviously you have setup an apparatus that allows the LLM to just shotgun attempts, whether it understands it or not. And sometimes it's not going to be able to find any solution. So it's not really appropriate for people to leave with the impression that anything measurable can be successfully optimized with LLMs.
I had this experience at work trying to optimize a little high level Pytorch. It can't really get better than it already was, but LLMs were quite willing to pretend they will. The real solution is I need to open a PR for one of Pytorch's tracking issues.
Yeah, but that's just the scientific process of hypothesis -> evidence -> conclusion.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
can LLMs write better code if you keep asking them to “write better code”?
One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.
I'm finding the bottleneck writing Rust code to be that compiling and running tests is too slow compared to TypeScript - which is unfortunate, because I want all of the guarantees of Rust - I just don't particularly care if it runs at the fastest speed possible - at least not at the prototype stage, or ever for a lot of what I'm doing (though I do want the guarantees regardless).
I wish it could run interpreted to massively speed up development, and that it had a Mull-like mutant testing framework so that mutants didn't take hours and tens-to-hundreds of gigabytes of space.
I ran into this exact issue and had Claude write a test run scheduler to avoid cargo process contention and migrated to nextest within it. I have --release build cycles on my local machine down to about three seconds from what was for the same project 60+ seconds before investing in the changes.
They're quite good at just iterating different "ideas" on a performance metric with an objective measure. They can use tools like `perf` and do some analysis on the output. Sometimes they go off in the weeds unproductively, and sometimes they give up because your goal was too high, but as long as you're sort of babysitting the process, you can make pretty rapid improvement to naive code.
"Rank the top findings/solutions by impact vs confidence" continues to be one of my best quickwins to add to all sorts of prompts. Bam, now you have a reasoned priority list.
I have found that having a separate agent (session / instance) do the benchmarking and reporting the results back to looping optimizing agent is a clean way to prevent cheating. The benchmarking agent has no reason to cheat, its goal is to just to run benchmarks when tickled.
I also found this is really nice for quality evals. Have one agent with no context on how something is made do a quality review, with lots of detailed feedback. Then pass back the review notes to the implementor for feedback. It works a lot better than having an agent self-evaluate its own quality.
As the author of Tera mentioned in the article, I am curious how it can get 2x faster. Was the benchmark using tera v2 with the `fast` feature enabled?
They were with tera 2.1.0, however using the default settings and not enabling the `fast` feature (which I did not know about, TIL).
The comparison benchmark results in the post were older and prior to further passes. After updating to tera 2.4.0 and enabling the `fast` feature set, I reran the benchmarks against the current codebase and the difference between my Rust crate and Tera is now about ~1.5x.
Some of the technical differences are (which in disclosure were surfaced by GPT 5.6 Sol): context and loop values are borrowed while Tera clones, cached static positions/lookups, less per-render heap setup, escaping chunks directly into the final String instead of bytes, and more granular bytecode functions which avoid stacks.
I keep the benchmark harness checked in and read-only to the agent, and let it touch everything else. Once it can't rewrite the measuring stick, the gaming just stops.
Not really. The LLM is the data model. The part doing the language manipulation. An agent is the combination of an LLM with an interface it knows how to use to take actions with, though it may just refer to the interface or connection or tool itself.
The LLM itself is primarily a transformer that handles tokenization, mapping, and neural-network reasoning (in simple terms). It's the engine. It takes in language, interprets it's meaning, and based on other language that supplies instructions, decides how to respond. The agent typically takes the form of an instruction for how to use a tool, combined with how to access it.
Technically, the chat session is itself an agentic tool because it's taking input and returning output through that interface. But it's kind of the default tool. Stdout, if you will.
Say you're in Gmail and you get a message that tells you about a meeting, but doesn't include an invite. If you tell Gemini to add an event for that meeting to your calendar, the LLM interprets your instruction, then uses an agent tool to read your email, interprets it, and then uses another agent tool to create a meeting on your Google Calendar.
Lot's of talk about how LLMs are bad at optimizing. And they are. They are still fundamentally semi-learned stochastic parrots. But LLMs are related to the Infinite Monkey Theorem - we've given monkeys typewriters, but pretrained the monkeys to be better than random.
They don't have to be good if they're cheap, directed, and sufficiently random to hit on something good within your budget.
Are LLMs better than our top mathematicians? No way! But they're able to spitball and falsify in parallel way faster. Eventually their directionally random output will hit something novel and interesting.
one thing nobody mentioned here, once the agent is looping against the same benchmark it will happily optimize for the benchmark itself and not the real workload. worth rerunning the win against a slightly different input shape after, just to check it did not memorize the harness instead of actually fixing anything.
I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
> Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
Speaking as somebody who is a performance geek, my knowledge came from relentless experimenting over the years (starting in the 8 bit era). Beyond the basics, I haven’t seen much on high-performance engineering online. To learn, you need to do the hard yards and I think performance tweaking becomes almost instinctive rather than something driven by a hard set of rules.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
I don't disagree, but IMO, a lot of code doesn't get to the point where those very low level techniques drive performance. Like, yes, if you are doing some heavy floating point math then that's where you end up needing it. However, in a lot of code finding hot paths and often simply switching out a O(n^2) for an O(n log n) or faster.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
I mean, sure, but it really does depend on what you're doing. If you're working on a library with collection-types and you want to make each iteration as fast as possible, then roll up your sleeves. If you're writing a compiler and you want your language's source-code to finish compiling this week, roll up the sleeves. If you're working on a game-engine and you want to draw more than everyone else, roll up the sleeves...
There are plenty of real-world reasons why you'd want to get knee deep in this stuff. I wasn't suggesting not using tools (I've literally spent the day buried in JetBrains' memory and tracing tools!), but those tools can only tell you what is happening now, not what to do to improve it.
Profiling is, of course, essential. But performance tweaking can be quite a laborious process: if you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course). Picking the correct data-structures is all in the 101 of performance engineering. That's in the literature. But it's all too basic and simplistic. Most performance minded engineers wouldn't need a profiling tool to know which data-structure to use.
At the smallest level there's a lot of mental theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
> but those tools can only tell you what is happening now, not what to do to improve it.
They tell you what's happening now, but they also tell you if what you've done has had a positive impact.
> If you're judging things by big-O notation, then that's a different level above the real low-level tweaking (imho of course).
I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems. Even in some of your examples like a compiler, game engine, or collection library, the big oh matters and if it's wrong, that can be a lot more important than shaving 0.1% on writing a function in a low level fashion. Big-Oh is gotten wrong a surprising amount of time even though it's 101 level stuff.
> At the smallest level there's a lot of theory building and experimentation as you try out different approaches, which is where the instinct and intuition starts to build. I never see any of that in discussions about performance engineering.
Oh because people get these things wrong all the time. That's why performance engineering stresses that you test, test, test and know what your testing and know why your testing could be wrong or corrupted. You should not trust your intuition because things change and it isn't always correct.
A good example of how easy it is to get measuring wrong. Imagine you start tweaking a function and you measure that your application became 1% faster. Was it the work you did on that function? Surprisingly, not always (at least not directly). Sometimes, it's the case that when you work on a function you re-align other functions as the machine code has to go it memory. It's possible that an undiscovered misaligned while loops was actually causing a large portion of your performance spill and by tweaking the function here, you aligned the while loop (or maybe a few of them). And, importantly, a new change somewhere else might re-unalign that same while loop.
You walk away thinking you've learn some low level lesson when in actuality your bit twiddling simply accidentally fixed something somewhere else.
This is why measuring is so important but also good measuring is even more important.
> I completely agree. My point isn't that Big-Oh is low level, but rather that Big-Oh is often enough for most programming problems.
For what I work on, the hidden constant is often more important than big O. For example, a hash map has better complexity than just searching through a vector. But if the vector is small enough it will best the hash nap for actual time. Just plain searching until you find the element will even beat binary search on a sorted vector for small enough vectors. The reasons are complex, to do with cache, prefetch, branch prediction and also just how many instructions your tight inner loop has. (And the specific reasons will vary between desktop class CPUs and microcontrollers. But both exhibit this pattern.)
You could argue that at that point why bother optimising at all (there aren't a lot of elements in the collection after all). But there are two distinct cases I have come across over the years where it still matters (and for what I work with, they represent the common cases):
* You need to look up in a small collection a lot (either lots of lookups into a few small collections or a few lookups each into lots of different small collections, I have seen both cases).
* Hard realtime code where predictable latency matters. Hashmap has a bad worst case, binary trees and binary searching has badly predictable memory access patterns. And in this case the collections are usually small anyway (there are only so many actuators and sensors your equipment has, and/or the embedded microcontroller doesn't have a lot of memory anyway).
> But if the vector is small enough it will best the hash nap for actual time.
This will all depend on the size and type of object stored in a vector.
If you have a relatively small and flat struct that you are storing, then sure that will likely win. But if you are working with a collection of pointers, then the hash map will (almost) always win.
> Hard realtime code where predictable latency matters. Hashmap has a bad worst case
The hash map worst case is a linear search. It will be a lookup + the search. This also depends on the implementation. You could, for example, use a robinhood hash map which trades insertion times for lookup times.
A bad hashmap implementation will store collisions in linked lists. A better one will try and put them in a b-tree. An even better one will use a vector for collisions. And the most cache friendly version stores everything in the table and does probing on collisions.
There are specific cases where vectors are better, but those are the exception and not the rule in my experience.
It all depends. I was thinking mainly of small keys that are at most a machine word. But yes I conceded that for large keys (extremely unusual for what I do, thus I didn't think of that case before) a hashmap wins.
Also, while on a desktop class CPU any pointer chasing you do tends to dominate, that is not the case on small and medium microcontrollers (which I work with a lot). They have very short pipelines and their memory is all internal SRAM most of the time. Here classic cycle counting is still king, so you need to ask yourself if it is quicker to compare the key than to calculate the hash and compare hashes.
Code size in general matters a lot on embedded. Even a simple hashmap will always have more code than a simple vector. And a state of the art implementation with probing and tombstones (such as swisstable in C++ and hashbrown in Rust) will be way too large to be usable.
> Even in some of your examples like a compiler, game engine, or collection library, the big oh matters
Sure, but for example - today - I am literally building and optimising high-performance collections for my open-source library. Big-O is irrelevant, because I have built pretty much all of the fundamental collection types, what I care about is lower than that: what happens when enumerating any one of those collection types. Big-O tells you the scale of the problem, but not the per-element cost, which is still important when you're building core data-structures.
I am concerned about cache-friendly memory-layouts, how to do collection compositions without unnecessary memory allocations, keeping enough guards in place to make the types safe whilst removing as many branches as possible, reducing memory copying as much as possible, and catching stupid shit the compiler or JIT does and try to work around it. Literally what happens per-instruction, per-iteration, not how to pick a big-O based data-structure: trying to make all data-structures as fast as possible.
Anyway, we seem to be talking past each other. You're talking about the basics, I'm talking about the original source of this thread which was that (apparently) LLMs are surprisingly bad at optimisation. Which, I am trying to highlight becomes almost voodoo at a low-enough level and that highly-optimised code looks progressively more strange and opaque (in the hunt for a few nanoseconds here and there), which for an LLM wouldn't look statistically significant. I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.
> I think the basics of data-structure choice should be easily within the realms of an LLM's current capabilities.
Surprisingly, it isn't. I catch the output of LLMs breaking these rules all the time. Just as it is pretty common in general programmer code.
The greatest sin I often find isn't necessarily Big-Oh related but rather multiple traversal problems. Much like regular programmers, LLMs love to do multiple passes over the same list to extract data. For example
A lot of programmers are oblivious to that sort of performance issue. It comes up a lot.I would think it's that the kinds of places which value this kind of knowledge often have major disincentive to share it. I'm thinking of HFT firms as one example.
Well if any labs want to scrape my knowledge of such things into model weights or faster training runs, I do consulting at movsoft.io :-)
It really is [as simple](https://github.com/ratatui/ratatui/pull/281#issuecomment-160...) as the Molly rocket video about terminals makes it seem; I am sure the models will catch up eventually. They have to overcome a lot of bad training data in most languages, but I find in Rust they usually do ok the first time.
Around six years ago, I worked for a switch manufacturer and we had insanely optimized networking code that will never see the public light of day.
I ran into the exact same wall when doing some SIMD + OpenMP optimisations.
Me: "Parallelise this loop without changing the results."
AI: "I can't do that, because of floating point accumulation."
Me: "So spin out a temporary fixed-length array, accumulate into that thread-locally, and then do an ordered summation of that at the end."
AI: "Oh, you're right!"
It's a fantastic tool for automating the implementation grind, but you still have to hand-feed it the strategy. I guess it will get there at some point.
Can you give it the valgrind suite to loop over? I have yet to try that with AI, but maybe the cachegrind tool is enough to help it.
A new meta is emerging of having high taste + infinite compute with little domain expertise. One recent example is Jarred Sumner taking a crack at the Riemann Hypothesis
> Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).
How much experience can you replace with infinite compute remains to be seen I guess.
https://news.ycombinator.com/item?id=49247070
I see the same thing, except I was working on high level performance reasoning. Whenever a piece of code has multiple steps that require multiple algorithms to work, AI almost always fails to guess which step is the slowest and what causes that step to be slow. Even Fable makes wrong guesses. You definitely need to give them a benchmark feedback loop.
Are there actually any humans who can reason about things like cache performance from first principles? I know there are some people who think they can, and I suspect they're fooling themselves. The one iron principle of micro-optimization at the level of cache hits is "measure, measure, measure", you just cannot think your way to the right answer on the first try. Processors and instruction sets today are too complicated, and tips that worked on one generation might be neutral or worse on the very next revision, making all the cargo cult knowledge passed around on this topic at best useless. I'll echo one of the sibling commenters here and say that LLMs probably bullshit their way to answers on questions like this because that's what humans online do as well.
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
Yes, people can demonstrably do this with high reliability.
Some humans carry detailed models of CPU microarchitectures in their heads, against which they can design code from first principles that will be nearly ideal on the first try. It is repeatable and verifiable. The best people can accurately predict the measured performance before writing a line of code.
Measurement is useful in cases where the model of software and hardware interaction is materially incomplete. In most cases this is because the people writing the software have insufficient understanding of the hardware. Having a limited understanding of the hardware is a choice.
It would be surprising if this wasn't possible.
I've been having great results with this kind of thing. I find that really just need a sensible framework within which the optimization can take place. Essentially just providing the measurement harness, and some sort of motivation for what I'm doing.
The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.
Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.
But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.
If it can be measured, then LLMs can optimize it.
Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.
And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.
AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.
Agree it's amazing how much low-hanging performance fruit AI can trivially find. On the other hand though, once you get through the obvious no-brainer stuff, there's a lot of non-trivial tradeoffs in performance and I think that still demands a good amount of expertise to guide the AI in the right direction. Obviously AI will continue working it's way up the value chain, but I think there's a glass ceiling for AI where the right macro tradeoffs and perspectives on how software should work will bump into the hard and often articulated reality that different stakeholders want different things and often have either magical thinking or even self-deception about how those desires can co-exist with what everyone else wants.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
Yeah, the biggest example is performance optimizations that sacrifice your data model to the point that you'd never accept them.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
I think they can be useful for quickly iterating through benchmarks and trying lots of ideas, but they won't come up with them on their own. Also, I'm not sure why, maybe some mean reversion thing, but they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything. Once I point it at a tool, it can be ok at using it (I say ok because they seem to skim the help docs, which is truly ironic, considering I seem to read it more thoroughly even though I'm 100x slower at it. I assume this is some token saving system prompt), but they won't suggest it for you.
This is why I'm not worried about being replaced for now or the forseeable future. For all of the improvements they've made, this part just never seems to change. They could slap another heuristic prompt for the edge case, but eventually it'll revert to the mean again.
I think there is a way to use LLMs to help with programming, but not when I'm not the driver in the seat writing the tests and deciding the architecture. Also I would never ship code written by them as the final product for anything I care about. Since I, like most people, find reading code to be arduous. The more fun thing to do is to force yourself to rewrite it all, treating the LLM's work as a rough draft.
> but [LLMs] won't come up with [ideas] on their own.
They can, in fact, generate plausible performance optimization ideas on their own.
> they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything
Make sure you process doesn't depend on anyone reading your mind.
When I run into things like this, it becomes a one-liner in my instructions/harness or in the canned prompt/skill I use that sets off a process.
In this case, I instruct agents to proactively build/improve diagnostic tooling if it would help them with their task + if it meets a bar of generalization/reusability (else it should be an ephemeral probe that gets abandoned at the end of the solution).
> If it can be measured, then LLMs can optimize it
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
The point of this post is that this is explicitly not the case. If the metric is measured, the agent finds a way eventually (around 5 total tries typically unless it gets stuck), and learns from iterations where changes caused a regression after a revert.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
I was actually emphasizing a methodology like yours. There is a lot more going on in your post than "the metric is measured." And even then, there's no guarantee the model will do anything except burn all your tokens trying stuff, it just won't commit the failed attempts.
"Attempting" implies a high risk of failure. That is not the case with the workflow described in the article, and from other comments, the approach is generally reliable as well.
It's also not terrible on token usage for smaller projects.
I don't know why I have to say this again, but I am telling you that the workflow you underlined in the article is good. But the more you argue about it, the more it sounds like you want people to take a more generalized conclusion than what you actually demonstrated.
> "Attempting" implies a high risk of failure.
Of course it does? Your safeguards also imply a high risk of failure. You have restrictions that just rollback everything the LLM "attempts" to do.
That is not to say that the overall workflow is failure prone, but obviously you have setup an apparatus that allows the LLM to just shotgun attempts, whether it understands it or not. And sometimes it's not going to be able to find any solution. So it's not really appropriate for people to leave with the impression that anything measurable can be successfully optimized with LLMs.
I had this experience at work trying to optimize a little high level Pytorch. It can't really get better than it already was, but LLMs were quite willing to pretend they will. The real solution is I need to open a PR for one of Pytorch's tracking issues.
Yeah, but that's just the scientific process of hypothesis -> evidence -> conclusion.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
> They can also spin round and round making the numbers worse
"Claude, if this idea doesn't measure as an improvement (use X benchmark and a T-test), discard it and try the next idea."
Yes, that is the spinning round and round part.
Crucially, it is easy to avoid making things worse, which was the entire thrust of my earlier comment and the part you've ignored in your response.
I didn't ignore it, I just think spinning round and round is also bad.
Ok. I think the goalposts have shifted from your original remarks.
can LLMs write better code if you keep asking them to “write better code”?
One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.
I've found that this is a good way to introduce bespoke code into the codebase whose perf gains don't generalize.
Unless its an easy memory/parallel/algorithmic win, its not worth it.
I'm finding the bottleneck writing Rust code to be that compiling and running tests is too slow compared to TypeScript - which is unfortunate, because I want all of the guarantees of Rust - I just don't particularly care if it runs at the fastest speed possible - at least not at the prototype stage, or ever for a lot of what I'm doing (though I do want the guarantees regardless).
I wish it could run interpreted to massively speed up development, and that it had a Mull-like mutant testing framework so that mutants didn't take hours and tens-to-hundreds of gigabytes of space.
I ran into this exact issue and had Claude write a test run scheduler to avoid cargo process contention and migrated to nextest within it. I have --release build cycles on my local machine down to about three seconds from what was for the same project 60+ seconds before investing in the changes.
They're quite good at just iterating different "ideas" on a performance metric with an objective measure. They can use tools like `perf` and do some analysis on the output. Sometimes they go off in the weeds unproductively, and sometimes they give up because your goal was too high, but as long as you're sort of babysitting the process, you can make pretty rapid improvement to naive code.
"Rank the top findings/solutions by impact vs confidence" continues to be one of my best quickwins to add to all sorts of prompts. Bam, now you have a reasoned priority list.
I have found that having a separate agent (session / instance) do the benchmarking and reporting the results back to looping optimizing agent is a clean way to prevent cheating. The benchmarking agent has no reason to cheat, its goal is to just to run benchmarks when tickled.
I also found this is really nice for quality evals. Have one agent with no context on how something is made do a quality review, with lots of detailed feedback. Then pass back the review notes to the implementor for feedback. It works a lot better than having an agent self-evaluate its own quality.
As the author of Tera mentioned in the article, I am curious how it can get 2x faster. Was the benchmark using tera v2 with the `fast` feature enabled?
They were with tera 2.1.0, however using the default settings and not enabling the `fast` feature (which I did not know about, TIL).
The comparison benchmark results in the post were older and prior to further passes. After updating to tera 2.4.0 and enabling the `fast` feature set, I reran the benchmarks against the current codebase and the difference between my Rust crate and Tera is now about ~1.5x.
Some of the technical differences are (which in disclosure were surfaced by GPT 5.6 Sol): context and loop values are borrowed while Tera clones, cached static positions/lookups, less per-render heap setup, escaping chunks directly into the final String instead of bytes, and more granular bytecode functions which avoid stacks.
Full type checking like in Askama and hot reloading of templates would be a great combination. Is that a feasible path with your project?
One could ask if the highly optimized code is still maintainable? Is adding new features still possible without performance regression?
On the other side in the age of coding agents code maintainability is not such an important metric anymore.
Yes, adding new features without regression is easy.
I keep the benchmark harness checked in and read-only to the agent, and let it touch everything else. Once it can't rewrite the measuring stick, the gaming just stops.
AI is heaven for bullshitters, bloggers and (fake) benchmark addicts.
[dead]
Tangent: are "LLM" and "Agent" synonymous?
Not really. The LLM is the data model. The part doing the language manipulation. An agent is the combination of an LLM with an interface it knows how to use to take actions with, though it may just refer to the interface or connection or tool itself.
The LLM itself is primarily a transformer that handles tokenization, mapping, and neural-network reasoning (in simple terms). It's the engine. It takes in language, interprets it's meaning, and based on other language that supplies instructions, decides how to respond. The agent typically takes the form of an instruction for how to use a tool, combined with how to access it.
Technically, the chat session is itself an agentic tool because it's taking input and returning output through that interface. But it's kind of the default tool. Stdout, if you will.
Say you're in Gmail and you get a message that tells you about a meeting, but doesn't include an invite. If you tell Gemini to add an event for that meeting to your calendar, the LLM interprets your instruction, then uses an agent tool to read your email, interprets it, and then uses another agent tool to create a meeting on your Google Calendar.
From an editing perspective, this went mostly down to a) attributing the change to a model rather than the harness and b) vibes.
In my mind no; an agent is an llm wrapped in more software (most trivially a while loop, but probably more then that)
Lot's of talk about how LLMs are bad at optimizing. And they are. They are still fundamentally semi-learned stochastic parrots. But LLMs are related to the Infinite Monkey Theorem - we've given monkeys typewriters, but pretrained the monkeys to be better than random.
They don't have to be good if they're cheap, directed, and sufficiently random to hit on something good within your budget.
Are LLMs better than our top mathematicians? No way! But they're able to spitball and falsify in parallel way faster. Eventually their directionally random output will hit something novel and interesting.
Agents are the ones doing the writing actually.
Now I'm imagining a JIT compliter runtime that relies on LLMs to find optimizations.
one thing nobody mentioned here, once the agent is looping against the same benchmark it will happily optimize for the benchmark itself and not the real workload. worth rerunning the win against a slightly different input shape after, just to check it did not memorize the harness instead of actually fixing anything.
Maybe LLM's should be integrated with the software, so they could optimize to the real hardware and workload ?
There is an entire paragraph about benchmaxxing and how to mitigate that.
[flagged]