The term harness engineering does not come from Birgitta Boeckeler's article in martinfowler.com. It was first used by Mitchel Hashimoto months earlier in his 'My AI adoption journey' blog post [1], and immediately picked up by OpenAI [2] a week later.
Also worth noting that this text is not only entirely AI generated, but meant for other agents to read, not us humans.
The site looks like aislop after skimming through 10 of his docs on the site, and the mistake you pointed out.
Something feels off, and claims on the site do not seem credible. Def not manually verified. If it was verified, the author did not go deep enough to give credits where it's due.
The only thing I got out of the site was to see terms I never heard of.
Honestly it would be helpful for someone to collect a list of these articles and concepts in a single location for engineers to get up to speed on these concepts. I think that is the only advantage of this AI generated website at the moment.
It’s fascinating to live through a the emergence of a new technology and to see people trying to make sense of it as they go.
I personally think that a lot of the words used around AI, harness, SKILLS, agents, RAG.. are make up words or close to it, words that do not have profound semantics, even though people are trying to, often after the fact, make sense of them.
It’s just popular words that are different enough that people like to use them to claim a new knowledge, or to market a product.
But we could use AI tooling instead of harness in the abstract, and it would be better to use more precise terms for more concrete use cases, agent loop, CLI, IDE..
It was a fun game at a time where papers were competing for attention, but now that it has become a proven technology, I hope we can find more precise and meaningful words.
The Agent/harness has no meaningful distinction. The backing model(s) can only read through a harness. Call it an agent, call it a tool, call it a library. They are all harnesses if they are backed by a model.
Broadly speaking,
- Copilot only reads AGENTS.md
- Codex reads AGENTS and CONTEXT
- Claude reads CLAUDE
This ai-literacy is noise, muddling the definitions and suggesting yet-another-md-file.
I think most people that use LLM coding tools enough independently derive most of the stuff in the article. For example my approach to garbage collection is to sample paths within the project and chunks of file content, then assemble context (similarity search results, git blame) for the agent to use to assess cruftiness and if found schedule cleanup.
definitely a little bit of shameless self promotion but I write about a lot of common harness concepts in the docs of Wingman (https://wingman.actor) - a client agnostic agent harness I've been working on for a while:
I think there is still plenty of innovation to be had in the space. Right now everything is still pretty bundled to the file system where getting skills and things to work inside of like a virtual worker requires simulating a virtual file system (usually). I believe even in the latest opencode v2 versions it still requires that a session be bundled to a working directory, I try to avoid (some of) that in Wingman.
What even is this "AI Literacy framework"? When I see such overengineering I look at what the author has done. In this case, I find the author runs a consultancy on engineering "Habitats for Humans and AI" (https://www.russmiles.com/), listing a bunch of books he did not write (the author names are conveniently cropped out). This doesn't even belong on LinkedIn.
I'm not the author -- but I am currently fortunate to be sitting in a workshop he is teaching on the topic and figured HN might like (especially since he gives a lot of his content out for free/OSS).
As someone who does loads of AI-driven dev and governance, I'm finding there are a lot of great nuggets here. Between him (chaos engineering) and Kent Beck (extreme programming) I'n a kid in the candy store and wanted to share.
Maybe code owners? I'm not sure if it'll do exactly what you want; but you can say that "such and such" files must be edited, or reviewed, by specific users.
Then, if someone opens a PR who isn't one of those people listed, it requires them to be a reviewer.
We use it for .editorconfig; specifically, we don't want a newcomer to come in, not like a style rule, and slip in a change.
The term harness engineering does not come from Birgitta Boeckeler's article in martinfowler.com. It was first used by Mitchel Hashimoto months earlier in his 'My AI adoption journey' blog post [1], and immediately picked up by OpenAI [2] a week later.
Also worth noting that this text is not only entirely AI generated, but meant for other agents to read, not us humans.
[1] https://mitchellh.com/writing/my-ai-adoption-journey
[2] https://openai.com/index/harness-engineering/
The site looks like aislop after skimming through 10 of his docs on the site, and the mistake you pointed out.
Something feels off, and claims on the site do not seem credible. Def not manually verified. If it was verified, the author did not go deep enough to give credits where it's due.
The only thing I got out of the site was to see terms I never heard of.
Honestly it would be helpful for someone to collect a list of these articles and concepts in a single location for engineers to get up to speed on these concepts. I think that is the only advantage of this AI generated website at the moment.
the one by https://earendil.com/posts/what-is-a-harness/ is also a good read
and if you want to see what a simple harness implementation looks like you can just dive into the code and ask your favorite agent about it
https://github.com/smol-env/smol
https://github.com/earendil-works/pi
Interesting: I initially confused smol for https://github.com/huggingface/smolagents
What they say about posting the wrong answer on social media is proven true here (though not my intention!). Thanks for the enlightenment!
It’s fascinating to live through a the emergence of a new technology and to see people trying to make sense of it as they go.
I personally think that a lot of the words used around AI, harness, SKILLS, agents, RAG.. are make up words or close to it, words that do not have profound semantics, even though people are trying to, often after the fact, make sense of them.
It’s just popular words that are different enough that people like to use them to claim a new knowledge, or to market a product.
But we could use AI tooling instead of harness in the abstract, and it would be better to use more precise terms for more concrete use cases, agent loop, CLI, IDE..
It was a fun game at a time where papers were competing for attention, but now that it has become a proven technology, I hope we can find more precise and meaningful words.
Hear hear! We techies are bad at naming things. "NoSQL" is probably a top contender there.
I give you "serverless" which is on someone else's... server.
It's lies all the way down...or up?
Indeed!
Here’s the original article this one is based on: https://martinfowler.com/articles/harness-engineering.html
And here is the original author on a podcast talking about it: https://open.spotify.com/episode/4FxEdjXldNhoh67KYVmbDu
Here's a better summary: https://github.com/rush86999/atom/blob/main/docs/architectur...
I also have a self-evolving architecture on it, but you have to be careful, especially if you're using multiple models (still WIP).
That looks like a “Deep Research” report generated by an LLM, not a summary.
Yeah, it is. Bad wording on my part, but the list is a summary of what's going on so far in agent harnesses.
This part seems relevant - they make explicit the difference between their HARNESS.md document and AGENTS.md. It's actually interesting: https://habitat-thinking.github.io/ai-literacy-superpowers/p...
The Agent/harness has no meaningful distinction. The backing model(s) can only read through a harness. Call it an agent, call it a tool, call it a library. They are all harnesses if they are backed by a model.
Broadly speaking,
- Copilot only reads AGENTS.md
- Codex reads AGENTS and CONTEXT
- Claude reads CLAUDE
This ai-literacy is noise, muddling the definitions and suggesting yet-another-md-file.
I put everything in CONTRIBUTING.md, and then put "See: CONTRIBUTING.md" in those vendor-specific files.
I just symlink each file to one master
I'm an AI Agent Harness .md File consultant, you can ask me anything
Not saying this is AI-gen but it's very dense and doesn't read like something I could glean info from easily
I think this article posted to HN last week did a better job explaining things IMO https://earendil.com/posts/what-is-a-harness/
> Not saying this is AI-gen
The prose is definitely written by a Claude-like
ai-literacy-superpowers is a set of plugins for agents, this text is not even meant for human consumption.
It is pretty obvious that Claude wrote it. No human writes README that long, and it's one of the contributors: https://github.com/Habitat-Thinking/ai-literacy-superpowers
This looks good - is there a version of it for CODEX? If not i guess it could easily be transformed for it.
Looks like typical material coming out of consulting companies like ThoughtWorks, designed to obfuscate rather than educate.
it lists 'Garbage Collection' as one of the 3 components of a harness
I don't know any harness that solves 'Garbage Collection' in the way described here
(most harnesses accelerate context pollution and code base drift via instructions they embed into system prompts, tool descriptions and skills)
I think most people that use LLM coding tools enough independently derive most of the stuff in the article. For example my approach to garbage collection is to sample paths within the project and chunks of file content, then assemble context (similarity search results, git blame) for the agent to use to assess cruftiness and if found schedule cleanup.
I like how agent review is a deprecation target. explicitly. Se then a harness matures by getting dumber and cheaper, not smarter.
definitely a little bit of shameless self promotion but I write about a lot of common harness concepts in the docs of Wingman (https://wingman.actor) - a client agnostic agent harness I've been working on for a while:
https://docs.wingman.actor/concepts/sessions/
I think there is still plenty of innovation to be had in the space. Right now everything is still pretty bundled to the file system where getting skills and things to work inside of like a virtual worker requires simulating a virtual file system (usually). I believe even in the latest opencode v2 versions it still requires that a session be bundled to a working directory, I try to avoid (some of) that in Wingman.
I also stumbled upon this site, which on a first glance looks really thorough. Interested to hear if folks have comments on it though:
https://walkinglabs.github.io/learn-harness-engineering/en/
Related currently:
The Harness is the Thing
https://news.ycombinator.com/item?id=49452346
What even is this "AI Literacy framework"? When I see such overengineering I look at what the author has done. In this case, I find the author runs a consultancy on engineering "Habitats for Humans and AI" (https://www.russmiles.com/), listing a bunch of books he did not write (the author names are conveniently cropped out). This doesn't even belong on LinkedIn.
I'm not the author -- but I am currently fortunate to be sitting in a workshop he is teaching on the topic and figured HN might like (especially since he gives a lot of his content out for free/OSS).
As someone who does loads of AI-driven dev and governance, I'm finding there are a lot of great nuggets here. Between him (chaos engineering) and Kent Beck (extreme programming) I'n a kid in the candy store and wanted to share.
What’s the best current practice if we want to enable agent code review and approval of GitHub PRs but only for specific users?
Maybe code owners? I'm not sure if it'll do exactly what you want; but you can say that "such and such" files must be edited, or reviewed, by specific users.
Then, if someone opens a PR who isn't one of those people listed, it requires them to be a reviewer.
We use it for .editorconfig; specifically, we don't want a newcomer to come in, not like a style rule, and slip in a change.
This is perfect, here's a tool for harness engineering to be more effective that I just released today https://github.com/issacnitin/RealDiff
Is Harness Engineering the new Prompt Engineering?
Yes it was.
"Explain the harness" is apparently the new "explain LLMs" genre of slop blogging, I now see articles to this effect on HN daily.
This is oddly reassuring
[dead]
[dead]