Loved this. It's funny how basic all of this can be. We have all these skills and and tricks and incantations whatever else, when it probably all just comes down to giving the agent the right info to set them up for success (just like you'd do with anyone you're delegating anything to).
I found it interesting that Thorsten mentions the importance of attaching screenshots.
In my experience, it varies widely by workload. Vision is great for some UI stuff but extremely terrible for my work, which involves visualizing connectivity and geometric stuff for road structure for autonomous vehicles. It can get super frustrating when I paste a screenshot and go "this polyline isn't connected when it should be" or "those polylines are self intersecting in an invalid way" and then the AI literally can't see what I'm talking about. It would have to then write debug code to dump out the coordinates of the polylines to convince itself that the bug even exists. Then it would often claim that it's fixed when it is clearly not.
I previously complained about GPT 5.6 Sol not being able to solve a simple "follow the lines" puzzle [1] without having to write custom Python code for it. GPT-6 Astra, however, is vastly improved, and could do it without writing Python. Still, for my work, it can occasionally struggle.
I think these AI platforms would guarantee bigger wins for themselves if they figured out how to give these models more of an object-conceptualization of their input than they currently have. Solving math puzzles is cool and all, but 99% of customers don't need to solve math puzzles like that; they're asking the model to answer questions about thins and to translate ideas into a real UI. A model that could reliably determine if/how its input/output isn't visually matching up to a design would be a big deal. Sure, we kind of have that now, but it's still in gimmick territory.
The thing about this is that in a different codebase, even just that search for the trumpet guy could either burn a ton of tokens and come back with nothing or find it but light the ocean on fire in the process. This is because trumpet guy would be named "asset-ds1f23nbiq35421fsda.jpg", and it would be located in an images folder with thousands of other similar assets. So the agent starts by grepping for reasonable names, discovers nothing but finds an image directory, and then searches inside. In the worst case, I guess it starts analyzing each image. Or maybe it checks the article like he mentions, but the codebase uses a component library and the filename thing is still true, so it analyzes every image in the page. This is a bit of hyperbole, of course. Please allow it for the sake of comedic example.
Many of his other prompt examples also illustrate this. The author is massively leveraging the architecture of the organization through familiarity. It could well be the case that even an LLM was used to set that up, but I doubt it happened in one prompt or without critical thought. To be fair, I guess he's sort of trying to point that out, but by saying "Look how lazy I am while it still works" or something like that in order to look cool, I think all he's conveying is "Look how easy this is", when it's potentially not and could increasingly cost him/his company wildly varying amounts of money based on how he goes about it. When tokens were virtually free, no one cared. This is increasingly not true. Many are drawing comparisons to early cloud architecture.
I find this is sort of similar to the problematic idea that in theory anyone can clone your repo and get a local dev environment running in a few seconds. So then you try it with a non-technical staff member and quickly realize how unrealistic that is as you run into abstraction after abstraction that is completely foreign to them.
Why is that problematic in the age of AI? Yes, previously, me being totally comfortable at a terminal where cd and ls are reflexive for me, working with someone non-technical, to get them to open a terminal and move around to type out git clone in order just to get started, was definitely a realization of a lot of things.
That was the past though. I've seen people who were good at computers but didn't know how to code, make some good programs where the AI wrote the code. Yeah, turns out tokens cost money, but they're still pretty cheap.
I observed this week that if you attach a specific counterexample to a rule, the agent follows it literally, whereas if you state only the principle without a counterexample, it follows the rule loosely. Instructions like "be honest" don't work, but instructions like "compare only based on semantic canonical form, because independent outputs have previously differed in hash values due to a single newline byte" do. In my experience, the true value of a prompt lies not in the number of sentences, but in the number of concrete examples attached to it.
This is where I think the value of Fable/Astra-class models shows, and maybe why some call these "AGI" (this transcript shows Fable as the target model, and I don't think this prompting strategy would work without it).
With Opus 4.x/GPT <=5.5, I was orchestrating "by hand", and everything required multiple individually-instructed and context'd steps. I saw real value in the Dark Factory/attractor/workflow graph-of-agent-roles pattern as a way to automate this .
With 5/5.6, prompts could be much larger, but needed to be very deep and wide - high level of detail, lots of context added, lots of examples. They could pull code practices and structure from what's present, -ish, but require lots of hand-holding, and subagent teams were just OK. The Dark Factory yielded to the Software Factory, where the graph didn't need to be pre-constructed - the agent could drive the workflow. Looked like the future to me.
With the newer class of models, I find that they are able to infer/derive much better and orchestrate and prompt subagents themselves. They are Driving. My prompts are still large, but only because they're doing more - the individual prompt-parts are much like what the transcript showed, the workflow has collapsed into just instructions.
I'm still dipping my toes in here, trying not to blow my token allocation too quickly, and even though astra and fable have major gaps I am seeing through the fog a bit. The Factory is kinda falling away into the bitter lesson I guess.
articles like this need to factor in the overall time/effort tradeoff in developing the set of prompts necessary to reach an overall goal... for example, if I can achieve the same results while being sloppy/lazy/vague with my prompts, I wouldn't bother with the extra discipline or time expended to craft "to spec" prompts
agreed. I'd also love to see more comparisons like this.
I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.
Slick presentation. His prompts look similar to my brain dumps. I’m in the habit of running these through Codex’s plan-mode first to ensure the AI is on the same page.
Loved this. It's funny how basic all of this can be. We have all these skills and and tricks and incantations whatever else, when it probably all just comes down to giving the agent the right info to set them up for success (just like you'd do with anyone you're delegating anything to).
I found it interesting that Thorsten mentions the importance of attaching screenshots.
In my experience, it varies widely by workload. Vision is great for some UI stuff but extremely terrible for my work, which involves visualizing connectivity and geometric stuff for road structure for autonomous vehicles. It can get super frustrating when I paste a screenshot and go "this polyline isn't connected when it should be" or "those polylines are self intersecting in an invalid way" and then the AI literally can't see what I'm talking about. It would have to then write debug code to dump out the coordinates of the polylines to convince itself that the bug even exists. Then it would often claim that it's fixed when it is clearly not.
I previously complained about GPT 5.6 Sol not being able to solve a simple "follow the lines" puzzle [1] without having to write custom Python code for it. GPT-6 Astra, however, is vastly improved, and could do it without writing Python. Still, for my work, it can occasionally struggle.
[1] https://news.ycombinator.com/item?id=49334444
I think these AI platforms would guarantee bigger wins for themselves if they figured out how to give these models more of an object-conceptualization of their input than they currently have. Solving math puzzles is cool and all, but 99% of customers don't need to solve math puzzles like that; they're asking the model to answer questions about thins and to translate ideas into a real UI. A model that could reliably determine if/how its input/output isn't visually matching up to a design would be a big deal. Sure, we kind of have that now, but it's still in gimmick territory.
The thing about this is that in a different codebase, even just that search for the trumpet guy could either burn a ton of tokens and come back with nothing or find it but light the ocean on fire in the process. This is because trumpet guy would be named "asset-ds1f23nbiq35421fsda.jpg", and it would be located in an images folder with thousands of other similar assets. So the agent starts by grepping for reasonable names, discovers nothing but finds an image directory, and then searches inside. In the worst case, I guess it starts analyzing each image. Or maybe it checks the article like he mentions, but the codebase uses a component library and the filename thing is still true, so it analyzes every image in the page. This is a bit of hyperbole, of course. Please allow it for the sake of comedic example.
Many of his other prompt examples also illustrate this. The author is massively leveraging the architecture of the organization through familiarity. It could well be the case that even an LLM was used to set that up, but I doubt it happened in one prompt or without critical thought. To be fair, I guess he's sort of trying to point that out, but by saying "Look how lazy I am while it still works" or something like that in order to look cool, I think all he's conveying is "Look how easy this is", when it's potentially not and could increasingly cost him/his company wildly varying amounts of money based on how he goes about it. When tokens were virtually free, no one cared. This is increasingly not true. Many are drawing comparisons to early cloud architecture.
I find this is sort of similar to the problematic idea that in theory anyone can clone your repo and get a local dev environment running in a few seconds. So then you try it with a non-technical staff member and quickly realize how unrealistic that is as you run into abstraction after abstraction that is completely foreign to them.
Why is that problematic in the age of AI? Yes, previously, me being totally comfortable at a terminal where cd and ls are reflexive for me, working with someone non-technical, to get them to open a terminal and move around to type out git clone in order just to get started, was definitely a realization of a lot of things.
That was the past though. I've seen people who were good at computers but didn't know how to code, make some good programs where the AI wrote the code. Yeah, turns out tokens cost money, but they're still pretty cheap.
I observed this week that if you attach a specific counterexample to a rule, the agent follows it literally, whereas if you state only the principle without a counterexample, it follows the rule loosely. Instructions like "be honest" don't work, but instructions like "compare only based on semantic canonical form, because independent outputs have previously differed in hash values due to a single newline byte" do. In my experience, the true value of a prompt lies not in the number of sentences, but in the number of concrete examples attached to it.
This is where I think the value of Fable/Astra-class models shows, and maybe why some call these "AGI" (this transcript shows Fable as the target model, and I don't think this prompting strategy would work without it).
With Opus 4.x/GPT <=5.5, I was orchestrating "by hand", and everything required multiple individually-instructed and context'd steps. I saw real value in the Dark Factory/attractor/workflow graph-of-agent-roles pattern as a way to automate this .
With 5/5.6, prompts could be much larger, but needed to be very deep and wide - high level of detail, lots of context added, lots of examples. They could pull code practices and structure from what's present, -ish, but require lots of hand-holding, and subagent teams were just OK. The Dark Factory yielded to the Software Factory, where the graph didn't need to be pre-constructed - the agent could drive the workflow. Looked like the future to me.
With the newer class of models, I find that they are able to infer/derive much better and orchestrate and prompt subagents themselves. They are Driving. My prompts are still large, but only because they're doing more - the individual prompt-parts are much like what the transcript showed, the workflow has collapsed into just instructions.
I'm still dipping my toes in here, trying not to blow my token allocation too quickly, and even though astra and fable have major gaps I am seeing through the fog a bit. The Factory is kinda falling away into the bitter lesson I guess.
I find that this is pretty different from how I prompt. I feel like AI use for greenfield projects is different from long-established codebases.
Try triangulating with multiple LLMs to synthesize information. The build by hand. The final composition will master understanding
articles like this need to factor in the overall time/effort tradeoff in developing the set of prompts necessary to reach an overall goal... for example, if I can achieve the same results while being sloppy/lazy/vague with my prompts, I wouldn't bother with the extra discipline or time expended to craft "to spec" prompts
agreed. I'd also love to see more comparisons like this.
I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.
Slick presentation. His prompts look similar to my brain dumps. I’m in the habit of running these through Codex’s plan-mode first to ensure the AI is on the same page.
layout is a bit rough on my mobile. really wished the text went below the video and the video went full width.
[dead]