I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
Is your main concern privacy? I've been using the zcode of and on for a few months now and it's definitely improved over what it was back in the spring. I also use DeepSeek's harness. Not sure which I prefer at this point. Used to be I preferred DSH, but zcode has some features I like over DSH.
I just looked it up: apparently they silently uploaded your entire repo with the entire git history to the cloud. A feature not controlled by any settings, operating and retrying silently.
Their excuse was that it was a file indexing optimisation and they forgot to create a toggle or setting around it. As an apology they open sourced the harness, but wiped out the history.
It sounds to me like a good distillation technique if the project has hundreds of commits written by Claude and detailed commit descriptions. Also for completeness Grok did the exact same thing two months ago.
Right! I had the exact opposite view when reading this comment. The good timeline is where one of (or perhaps a small number of) the open source harnesses becomes so dominant that the models compete to have the model that is the best trained to work with that harness. (I'm hoping this would be Pi, because it's my favorite, but mostly I just want it to be some model-agnostic open source harness that wins.)
Realistically, it's never going to be Pi, for the same reason that video game consoles are always more performant than PCs for a given hardware spec.
There's no canonical Pi harness, it's too variable. There might be a world where something Pi based is the target, like OMP, but then you have to hesitate when you start extending the harness because you don't know what effect any given extension will have on model performance.
It won't be Pi because there's really no singular Pi that is broadly useful without explicit configuration of plugins. Pi requires plugins to do lots of things that are OOB on other harnesses that people care about. MCP, OpenTelemetry, etc. It may be some offshoot or something built on top of Pi that is more standardized, but it won't be Pi.
MiMo _needed_ it. MiMo 2.5 Pro really struggled when given more tools, causing it to lose other capabilities. e.g. in security benchmarks I was doing, most of the medium-to-large models got better or stayed roughly the same at finding security vulnerabilities when given more tools (e.g. treesitter, semgrep, a full bash/python environment, etc.) vs. when only given the ability to read the files in the repo. But, MiMo got notably worse. It seemed to get confused by all the options. MiMo 2.5 Pro just reading files is an excellent security bug finder, at the pareto frontier at the time (cheapest option to find as many bugs as Opus 4.8, which was current at the time), but adding tools cratered it down to small model territory.
I haven't tested the new version, and I still have a bunch of tokens on my token plan, so I might give 2.6 a go in some similar tests to see if it still has the analysis paralysis problem of 2.5.
This is the very reason why I avoid to use native harness. They are optimized for the economic benefit of the provider, not for mine. Love Pi because it does a good job managing the context, open code meh, claude code nope.
This matters less as models get better and everyone settles on the same overall harness architectures. The model matters more than the harness anyway.
The bigger issue is that the use cases and harnesses for models is infinite, which is hard to compress into benchmark numbers that actually apply to you.
Everyone is benchmaxxing, desperate to sell, and almost nobody except the labs is doing actual science on the results, so harnesses tend to be chosen on voodoo and hunches, like which company made it. There isn't necessarily a good alternative though, bearing the cost of being a harness researcher is probably not many people's goal.
> The model matters more than the harness anyway
>
> Everyone is benchmaxxing
>
> ...harnesses tend to be chosen on voodoo and hunches...
I get what you're saying, but their graphic on performance here uses the exact same model with different harnesses and definitively shows that there is a significant difference in both accuracy and cost. The whole point of their technical implementation and design decision here is to highlight that it's not "voodoo and hunches", but observable data.
Fable 5 on Claude Code scored 61.8% at a cost of $248.05 while Fable 5 on OpenCode beat it at 66.3% at $73.42. The same model, the same benchmark; only the harness is different with a ~5 point difference in accuracy while costing significantly less. So if we are to believe the author and these results are repeatable, then it would seem that the harness matters.
The point of this framing here is specifically to address 1) benchmaxxing by using the same model, 2) NOT choose a harness on "voodoo and hunches" by using actual data to back the assertions. Your comment feels misguided and completely hand waves the actual data points here.
Agreed, especially since the more frontier models are able to accomplish in a vacuum, the more people will trust them. That being said, tool use is still really important for pulling in the right information.
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.
There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
It's a fair point, but for any well-established capability, the plugin with the most monthly downloads (overwhelmingly) is generally going to be "the one". For MCP, the top plugin [1] has 1M/mo, and the next one down has less than 2k/mo.
I do agree that in the case of benchmarking, it's a bit silly to put the onus on the benchmark runner to try and pick the right plugin for each capability that won't make someone point to /that/ as the reason Pi didn't do as well as expected.
> There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark?
The one that everyone else uses, the most popular. Exactly the one I listed.
I wrote "pi install npm:pi-mcp-adapter". It would install the mcp adapter.
> Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
That's the whole point of Pi, it doesn't have to reach checklist level parity with Strand. Just benchmark it with its default config. Add mcp adapter if your benchmark really requires it, but that's it.
> It’s like you didn’t even read my comment.
I did, but it seems you didn't read my comment, or you didn't understand that the command I gave you is an actual recipe, and not a pseudo instruction.
OhMyPi is essentially Claude like bloated and Pi is so barebones that many will just get frustrated trying to add bricks after bricks to make it usable for complex workflows - so I don't really see any point of it being in such benchmarks. Pi should have added option to easily add "packs" (and then also tweak tehm) sort of things. When I tried it I actually thought that's what OMP will be, but it was not. It was a different harness altogether in a way and not a nice way.
I wish we could use just the frontend of omp harness-agnostically, like the provider switching, stats dashboard, subscription pooling etc. are excellent features, but it doesn't seem inconceivable that we can have all that with a swappable "actual" harness. I've tried to make omp more pi-like by lazy loading most of the tools rather than dumping them into system prompt. haven't benchmarked this yet.
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses
> With Fable 5, Strands harness cost 77% less than Claude Code and scored higher on Terminal Bench 2.1.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
The scores are in blog post's bar chart. For Terminal Bench 2.1, Strands harness (Fable 5) scored 69.7 while Claude Code (Fable 5) scored 61.8. This is on high effort.
I hear you tho about saturation. We're working on a follow-up deep dive post with more harnesses, so could look into Terminal Bench 4.0?
I remember reading about strands SDK and it looked great in terms how everything is an event that you can extend, so this harness feels quite about right.
However, for this kind of customisation, Pi is actually quite great. One of the most things I love about Pi is ability to ask it to create an extension and it does it quite well as it’s part of their docs. Also ability to customise the system prompt to avoid the clutter that Claude Code add (around 20k system prompt that mostly had nothing to do with the code).
The demo was showing something I have created for my Pi setup, which is asking me in each new session which skills and MCP I would to enable for the session. This works quite well if you have multiple projects where you don’t need all skills but just a small subset
> But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling.
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
Yes, it’s a wild claim. I built a harness for a random side project without even thinking hard about it. The harness was that from the hardest part of the project.
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
I am using Pi with anthropic models for plan & execution, openai models for code review and fallback in case I run out of tokens. Also have custom hooks to automatically compact, switch models based on the type of tasks, change effort and custom statusline. Also have custom setup in Pi to calls tools and CLIs instead of relying on mcps. The level of customization I can reach with pi is not really feasiable with codex or claudeCode. By the way anthropic changed it's ToS regarding custom harness: you can`t use their model unless you got extra usage enabled or paying through APi billing, you cant use your subscription anymore. We all guessed what the reason might be. Also codex is open source, I havent really looked at that project yet because Pi fulfill my desires for now.
I'm using Opencode Go in OMP or Hermes. $10 a month and I have only ever hit a limit using qwen3.8MAX on X-High. This is a migration from 20x on Claude.
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
I've been going the opposite direction, making opencode plugins for features I sketched out in my custom harness. I may end this and go back to my own, opencode is so sloppy, but then what harness isn't?
Including Ollama support is the new token "open option", like a cross-platform application that claims to support Windows, macOS, and Linux, and targets Ubuntu 16.04 only. Just target the OpenAI-compatible API with custom endpoints, it's not that hard. https://schizo.cooking/schizo-takes/9.html
Man, I'm sorry for the lack of original thought, but I am so sick of AI websites.
I would seriously rather see a plain unstyled html page with a marquee tag saying "A HUMAN WROTE THE NEXT THREE PARAGRAPHS"
I switch between Pi, Autolith, and my own harness. I come to this website. I don't know what the hell a strand agent is. I don't know who is putting this out. I don't understand whether I'm meant to use a GUI or write code in order to configure it. All the screenshots show a GUI. "use one line of python or javascript to configure." I don't know if this is a harness, an orchestrator, an agent framework, or all of the above. I think it's hard. People get so caught up in making their own thing - they lose track of the state of the art. I have no idea what the value prop for Strands Harness is. But maybe I'm not the target audience.
Also, I don't really care about 30% token efficiency gains. Maybe I should. But this is not my experience. Don't get me wrong, I'm conservative with tokens. But I also have a LOT of convenience / essentials which bloat up context. And I'd rather bloat context and have it work well than not have it work well. And when GLM 5.3 Flash came out - all of a sudden I had more tokens than I knew what to do with. That's when I started using OpenClaw type agents for the first time.
I don't have the tokens/cash to throw at something like Gas Town or OpenRig. And I'm still not at all convinced that those software factories are actually the future of everything. Maybe so. But I just don't see why I would invest the time to learn a piece of software without really understanding the value prop. It's crazy to say, but it's just not enough anymore for software to be free. Because the cost of me implementing my own software is rapidly approaching free. And I like mine better.
"Oh, well I just need something that works out of the box." Wait, why? What does it need to work for? For making software? lol.
I'm more and more hesitant to use third party harnesses.
It's so easy to take opencode or pi and build my own for exactly what I want that I'm seeing projects like this to be noisy and less valuable over time.
I wish this wasn't the case. I love software, but it's becoming increasingly easy to just build my own.
finally paying up for chatGPT and using the Codex desktop app was my real "I'm sold" moment with AI.
Setting up projects and working with the AI on local files has been great, but only for my personal account. I've been trying to get it set up for work that provides OpenAI models through a 3rd party tool, company hosted models, as local-machine models and the UX is just straight up awful.
there's no GUI for profiles or custom endpoints, the config.toml sucks and the overall experience is primitive.
At this stage I really want a Codex-like harness but I need more fluid control over the models, I want features like pinning a project to a provider, as well as pulling in all the models from that provider, also having all providers available.
So if I need to pop over to one project to consult about product A, then pop to another project to do some code analysis on product B I can do so fluidly and have my tokens billed to the right place for each concern.
or a project that can span all of the resources. like having the OpenAi models orchestrate sub-agents on the local or hosted models.
Because vanilla pi doesn't do a lot of important things out of the box, and they didn't want to open the "which extensions should we add" can of worms.
it's less about the interfaces and more that it feels disingenuous to compare token usage to a pi setup that injects a bunch of stuff into context rather than the base version. it makes me think they tested it with pi, found they couldnt meaningfully beat its token usage + pass rate, and omitted the comparison. i'd love to be proven wrong, but this seems like the easy explanation
Their highest starred repo, the SDK, has 7.8K stars. Yet they felt the need to show 13000+ on this landing page (summing up all repo stars). Technically correct inflating it? Well, that's confidence boosting.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
Hi I'm Albert Zhao, one of the blog authors. Strands harness is meant to be deployed outside of AWS too. I got it running on Cloudflare Containers and Modal before this launch (deployment docs coming soon). We just want to keep making a really good harness, esp since Strands is born from a lot of internal headbanging around building agents
Yeah Strands powers thousands of agents at Amazon, so we all really benefit from being open source thanks to improvements/github issues from the community. Our focus is making agents run easily and reliably in prod
1. Any tool result longer than 1500 tokens gets a link instead.
2. Forced compaction at 85%, removing all past messages apart from the last 4?
I'm sure these guys made it look good on the benchmarks, but the frontier labs have been hill-climbing this game for a while now, so odds are this just doesn't work for long-horizon tasks labs are optimizing for.
so far I have seen many harnesses that are built to provide similar quality to the provider's harness for lower cost. Has there been any projects that prioritizes quality and quality consistency?
is anyone using strands ? I looked into it and ended up picking mastra as it had a better story / seems more complete for people making agents in typescript.
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
Beyond that though, I'm certainly interested in the performance side of things. "Keep an eye out for a follow-up paper from our researchers regarding these benchmarks." Yes plz.
Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
I am increasingly hesitant to use non-native harnesses - model providers are now starting to train their agents for use within the harness. An eval like terminal bench can only capture so much data. I don't want to have to assess each harness every model release to make sure it's working as well as it can.
I mostly use GLM. But I will not touch ZAI's harness with a 10 mile long pole no matter how efficiently they couple it with GLM.
Is your main concern privacy? I've been using the zcode of and on for a few months now and it's definitely improved over what it was back in the spring. I also use DeepSeek's harness. Not sure which I prefer at this point. Used to be I preferred DSH, but zcode has some features I like over DSH.
Did you miss the whole fiasco from last week?
It's why I asked if your main concern was privacy (data leakage). They open-sourced their harness this week so you can always audit it.
I seemingly did - what happened?
I just looked it up: apparently they silently uploaded your entire repo with the entire git history to the cloud. A feature not controlled by any settings, operating and retrying silently.
Their excuse was that it was a file indexing optimisation and they forgot to create a toggle or setting around it. As an apology they open sourced the harness, but wiped out the history.
It sounds to me like a good distillation technique if the project has hundreds of commits written by Claude and detailed commit descriptions. Also for completeness Grok did the exact same thing two months ago.
https://www.theregister.com/security/2026/09/22/zai-says-sor...
I’m the opposite. I want one open source harness to rule them all
Cost efficiency is a plus
Right! I had the exact opposite view when reading this comment. The good timeline is where one of (or perhaps a small number of) the open source harnesses becomes so dominant that the models compete to have the model that is the best trained to work with that harness. (I'm hoping this would be Pi, because it's my favorite, but mostly I just want it to be some model-agnostic open source harness that wins.)
Realistically, it's never going to be Pi, for the same reason that video game consoles are always more performant than PCs for a given hardware spec.
There's no canonical Pi harness, it's too variable. There might be a world where something Pi based is the target, like OMP, but then you have to hesitate when you start extending the harness because you don't know what effect any given extension will have on model performance.
Yeah this does seem right. Oh well.
It won't be Pi because there's really no singular Pi that is broadly useful without explicit configuration of plugins. Pi requires plugins to do lots of things that are OOB on other harnesses that people care about. MCP, OpenTelemetry, etc. It may be some offshoot or something built on top of Pi that is more standardized, but it won't be Pi.
Yes, definitely fair. Ideally all the models would be trained against "Pi in exactly the configuration I personally prefer", but alas :)
Interestingly, MiMo was trained across multiple harnesses and it improved capabilities
https://mimo.mi.com/docs/en-US/news/latest/v2-6
MiMo _needed_ it. MiMo 2.5 Pro really struggled when given more tools, causing it to lose other capabilities. e.g. in security benchmarks I was doing, most of the medium-to-large models got better or stayed roughly the same at finding security vulnerabilities when given more tools (e.g. treesitter, semgrep, a full bash/python environment, etc.) vs. when only given the ability to read the files in the repo. But, MiMo got notably worse. It seemed to get confused by all the options. MiMo 2.5 Pro just reading files is an excellent security bug finder, at the pareto frontier at the time (cheapest option to find as many bugs as Opus 4.8, which was current at the time), but adding tools cratered it down to small model territory.
I haven't tested the new version, and I still have a bunch of tokens on my token plan, so I might give 2.6 a go in some similar tests to see if it still has the analysis paralysis problem of 2.5.
Mimo 2.5 and mimo 2.5 pro are completely different beasts.
The non-pro version is mostly useless
I only tested MiMo 2.5 Pro. I have updated my comment to make that clear in the first mention.
This is the very reason why I avoid to use native harness. They are optimized for the economic benefit of the provider, not for mine. Love Pi because it does a good job managing the context, open code meh, claude code nope.
This matters less as models get better and everyone settles on the same overall harness architectures. The model matters more than the harness anyway.
The bigger issue is that the use cases and harnesses for models is infinite, which is hard to compress into benchmark numbers that actually apply to you.
Everyone is benchmaxxing, desperate to sell, and almost nobody except the labs is doing actual science on the results, so harnesses tend to be chosen on voodoo and hunches, like which company made it. There isn't necessarily a good alternative though, bearing the cost of being a harness researcher is probably not many people's goal.
Fable 5 on Claude Code scored 61.8% at a cost of $248.05 while Fable 5 on OpenCode beat it at 66.3% at $73.42. The same model, the same benchmark; only the harness is different with a ~5 point difference in accuracy while costing significantly less. So if we are to believe the author and these results are repeatable, then it would seem that the harness matters.
The point of this framing here is specifically to address 1) benchmaxxing by using the same model, 2) NOT choose a harness on "voodoo and hunches" by using actual data to back the assertions. Your comment feels misguided and completely hand waves the actual data points here.
Agreed, especially since the more frontier models are able to accomplish in a vacuum, the more people will trust them. That being said, tool use is still really important for pulling in the right information.
And yet, open harnesses works better and use less tokens than native harnesses trained to consume as much tokens as possible.
Why is Pi not in the benchmarks? Deepseek beats Strands and its built on Pi so that’s all I needed to know.
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.
If "pi install npm:pi-mcp-adapter" is preventing them from including pi in benchmarks, then they might not be competent to run trustworthy benchmarks.
There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
It’s like you didn’t even read my comment.
It's a fair point, but for any well-established capability, the plugin with the most monthly downloads (overwhelmingly) is generally going to be "the one". For MCP, the top plugin [1] has 1M/mo, and the next one down has less than 2k/mo.
I do agree that in the case of benchmarking, it's a bit silly to put the onus on the benchmark runner to try and pick the right plugin for each capability that won't make someone point to /that/ as the reason Pi didn't do as well as expected.
[1] https://pi.dev/packages/pi-mcp-adapter?name=pi-mcp-adapter
> There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark?
The one that everyone else uses, the most popular. Exactly the one I listed.
I wrote "pi install npm:pi-mcp-adapter". It would install the mcp adapter.
> Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
That's the whole point of Pi, it doesn't have to reach checklist level parity with Strand. Just benchmark it with its default config. Add mcp adapter if your benchmark really requires it, but that's it.
> It’s like you didn’t even read my comment.
I did, but it seems you didn't read my comment, or you didn't understand that the command I gave you is an actual recipe, and not a pseudo instruction.
Deepseek harness is not actually built on Pi harness. It's an independent project.
This is true, but the team said on Twitter that they were heavily inspired by Pi and this whole team used it extensively:
https://x.com/tianyi/status/2088306143772946499
Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.
That’s fair, but Strands is advertising their harness needing way less tokens which does relate to cost.
In that same vein, Pi is less bloated then Deepseek and Oh My Pi, which are built on top of Pi. Isn’t it dubious to leave it out?
OhMyPi is essentially Claude like bloated and Pi is so barebones that many will just get frustrated trying to add bricks after bricks to make it usable for complex workflows - so I don't really see any point of it being in such benchmarks. Pi should have added option to easily add "packs" (and then also tweak tehm) sort of things. When I tried it I actually thought that's what OMP will be, but it was not. It was a different harness altogether in a way and not a nice way.
I wish we could use just the frontend of omp harness-agnostically, like the provider switching, stats dashboard, subscription pooling etc. are excellent features, but it doesn't seem inconceivable that we can have all that with a swappable "actual" harness. I've tried to make omp more pi-like by lazy loading most of the tools rather than dumping them into system prompt. haven't benchmarked this yet.
Do you have any write up on that - how you went about it? Also, in that case why not just make Pi slightly OMP like? Really curious.
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses
> With Fable 5, Strands harness cost 77% less than Claude Code and scored higher on Terminal Bench 2.1.
Terminal Bench 2.1 is saturated. Many token saving techniques would save money and score basically the same running Fable 5 against Terminal Bench 2.1. (They claim a better score but don’t say how much better. I’d bet my favorite hat that it’s not statistically significant.)
This is at least the fourth time I’ve seen a project hit front page with a “save money with same score on saturated benchmark” claim.
The scores are in blog post's bar chart. For Terminal Bench 2.1, Strands harness (Fable 5) scored 69.7 while Claude Code (Fable 5) scored 61.8. This is on high effort.
I hear you tho about saturation. We're working on a follow-up deep dive post with more harnesses, so could look into Terminal Bench 4.0?
Then I'm really confused. Terminal Bench 2.1 scores on Artificial Analysis are like 80-90%. https://artificialanalysis.ai/evaluations/terminalbench-2-1
AA isn’t testing different harnesses there, they’re testing different models on the same common harness:
“We run Terminal-Bench 2.1 with the Terminus 2 agent harness in an e2b sandbox and report pass@1 averaged over 3 repeats per task.”
This is a different agent harness than those Strands was testing with, and there’s no guarantee the pass criteria matches up the same either.
I remember reading about strands SDK and it looked great in terms how everything is an event that you can extend, so this harness feels quite about right.
However, for this kind of customisation, Pi is actually quite great. One of the most things I love about Pi is ability to ask it to create an extension and it does it quite well as it’s part of their docs. Also ability to customise the system prompt to avoid the clutter that Claude Code add (around 20k system prompt that mostly had nothing to do with the code).
The demo was showing something I have created for my Pi setup, which is asking me in each new session which skills and MCP I would to enable for the session. This works quite well if you have multiple projects where you don’t need all skills but just a small subset
> But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling.
It's wild to me to claim that it's tricky to customize one of these harnesses and for that to be the entire justification for an entirely different harness.
It's really not that hard. If you want to reduce costs then all you need to do is practice delegation: instead of using the strong model, all the time to do everything, instead, you have the stronger model delegate well-defined tasks to a weaker model. Patterns like these are really easy to wire up.
Yes, it’s a wild claim. I built a harness for a random side project without even thinking hard about it. The harness was that from the hardest part of the project.
Is this corporate confabulation?
How are people using custom harnesses cost effectively? Do they avoid Anthropic models so they can use OpenAI subscription pricing and open weights stuff?
For what it's worth I've been using Anthropic models on Pi for months now with no issues. It's not recommended since it breaks TOS but you can do it.
You mean through the API? I don't see how to do this with subscription billing.
I think with things like this. https://github.com/gotgenes/pi-anthropic-auth
I haven’t tested it though
I use Pi and mostly open weight models. I pay for the $20/month Ollama plan and use Deepseek and GLM through that. I’ve never hit the limits on it, but I tend to ask for targeted things rather than “implement a whole feature in one prompt”.
I do keep an OpenRouter account topped up for things that Ollama doesn’t have. 99% of my usage there is embeddings, the other 1% is wanting to test some new model Ollama doesn’t have.
That’s exactly what I do. OpenAI + Opencode Go subs, 0 interest in Claude.
I haven't used the US labs in months. Kimik3 for planning and GLM5.3flash or the new Deepseek Flash for code.
Yes. Or- use them at work, where management is taking a... hands off approach to ~integrating ai~ into the workplace.
OpenAI. It avoids Claudish too.
I am using Pi with anthropic models for plan & execution, openai models for code review and fallback in case I run out of tokens. Also have custom hooks to automatically compact, switch models based on the type of tasks, change effort and custom statusline. Also have custom setup in Pi to calls tools and CLIs instead of relying on mcps. The level of customization I can reach with pi is not really feasiable with codex or claudeCode. By the way anthropic changed it's ToS regarding custom harness: you can`t use their model unless you got extra usage enabled or paying through APi billing, you cant use your subscription anymore. We all guessed what the reason might be. Also codex is open source, I havent really looked at that project yet because Pi fulfill my desires for now.
Sounds wildly overcomplicated.
I'm using Opencode Go in OMP or Hermes. $10 a month and I have only ever hit a limit using qwen3.8MAX on X-High. This is a migration from 20x on Claude.
pi on open weights, I only use frontier models to do a review pass
Does anyone else take these kinds of articles, drop them into ChatGPT, crank it up to Pro, and then have it write issues against your personal harness?
It's tempting, but I still use Claude Code because it's required as part of the subscription and I want the discount. What models do you use?
any chance you have a public git repo for that personal harness of yours? curious minds want to know.
I've been going the opposite direction, making opencode plugins for features I sketched out in my custom harness. I may end this and go back to my own, opencode is so sloppy, but then what harness isn't?
Can you also compare in the charts https://maki.sh?
Should give you some competition.
Including Ollama support is the new token "open option", like a cross-platform application that claims to support Windows, macOS, and Linux, and targets Ubuntu 16.04 only. Just target the OpenAI-compatible API with custom endpoints, it's not that hard. https://schizo.cooking/schizo-takes/9.html
Man, I'm sorry for the lack of original thought, but I am so sick of AI websites.
I would seriously rather see a plain unstyled html page with a marquee tag saying "A HUMAN WROTE THE NEXT THREE PARAGRAPHS"
I switch between Pi, Autolith, and my own harness. I come to this website. I don't know what the hell a strand agent is. I don't know who is putting this out. I don't understand whether I'm meant to use a GUI or write code in order to configure it. All the screenshots show a GUI. "use one line of python or javascript to configure." I don't know if this is a harness, an orchestrator, an agent framework, or all of the above. I think it's hard. People get so caught up in making their own thing - they lose track of the state of the art. I have no idea what the value prop for Strands Harness is. But maybe I'm not the target audience.
Also, I don't really care about 30% token efficiency gains. Maybe I should. But this is not my experience. Don't get me wrong, I'm conservative with tokens. But I also have a LOT of convenience / essentials which bloat up context. And I'd rather bloat context and have it work well than not have it work well. And when GLM 5.3 Flash came out - all of a sudden I had more tokens than I knew what to do with. That's when I started using OpenClaw type agents for the first time.
I don't have the tokens/cash to throw at something like Gas Town or OpenRig. And I'm still not at all convinced that those software factories are actually the future of everything. Maybe so. But I just don't see why I would invest the time to learn a piece of software without really understanding the value prop. It's crazy to say, but it's just not enough anymore for software to be free. Because the cost of me implementing my own software is rapidly approaching free. And I like mine better.
"Oh, well I just need something that works out of the box." Wait, why? What does it need to work for? For making software? lol.
The interesting thing about this, if you think about it, is that this is an AWS OSS release.
I'm more and more hesitant to use third party harnesses. It's so easy to take opencode or pi and build my own for exactly what I want that I'm seeing projects like this to be noisy and less valuable over time.
I wish this wasn't the case. I love software, but it's becoming increasingly easy to just build my own.
Tell your harness to fix overflow in header between 800-1400px.
finally paying up for chatGPT and using the Codex desktop app was my real "I'm sold" moment with AI.
Setting up projects and working with the AI on local files has been great, but only for my personal account. I've been trying to get it set up for work that provides OpenAI models through a 3rd party tool, company hosted models, as local-machine models and the UX is just straight up awful.
there's no GUI for profiles or custom endpoints, the config.toml sucks and the overall experience is primitive.
At this stage I really want a Codex-like harness but I need more fluid control over the models, I want features like pinning a project to a provider, as well as pulling in all the models from that provider, also having all providers available.
So if I need to pop over to one project to consult about product A, then pop to another project to do some code analysis on product B I can do so fluidly and have my tokens billed to the right place for each concern.
or a project that can span all of the resources. like having the OpenAi models orchestrate sub-agents on the local or hosted models.
This seems to be from AWS team. Is that right?
Correct the Strands team and I are a part of AWS, though all of our work is open source. We want folks to deploy Strands harness anywhere they want
why would you include oh-my-pi in the comparison but not vanilla pi?
Because vanilla pi doesn't do a lot of important things out of the box, and they didn't want to open the "which extensions should we add" can of worms.
if anything, using oh-my-pi opened that can more than using pi (or better: both!) would have
Am I wrong in saying that the interfaces presented to the model in OMP versus plain old Pi are identical?
OMP has a lot of candy that raises token cost compared to vanilla pi
it's less about the interfaces and more that it feels disingenuous to compare token usage to a pi setup that injects a bunch of stuff into context rather than the base version. it makes me think they tested it with pi, found they couldnt meaningfully beat its token usage + pass rate, and omitted the comparison. i'd love to be proven wrong, but this seems like the easy explanation
I’m sure it was not a coincidence that this was released the day after Kimi 3 was added to bedrock.
Their highest starred repo, the SDK, has 7.8K stars. Yet they felt the need to show 13000+ on this landing page (summing up all repo stars). Technically correct inflating it? Well, that's confidence boosting.
Supported provider list is nowhere close to pi's or opencode's
Will they block me if I build an agent with this that shops on Amazon?
Is strands the only option if you're trying to scale up agents? Thinking 1000s of agents hosted on AWS / other cloud providers.
I think LangChain is a popular alternative in the space
Strands is AWS.
Amazon is so hopelessly behind in AI, nothing they produce aside from cloud infrastructure is actually good
The big threat to AWS is that coding agents dont need all of their complicated infrastructure, which was built for humans. Agents can use low level primitives, i.e. just a raw server
Hi I'm Albert Zhao, one of the blog authors. Strands harness is meant to be deployed outside of AWS too. I got it running on Cloudflare Containers and Modal before this launch (deployment docs coming soon). We just want to keep making a really good harness, esp since Strands is born from a lot of internal headbanging around building agents
I believe that you just want to make a really good harness, but is that all amazon wants it to be or do they want more?
Yeah Strands powers thousands of agents at Amazon, so we all really benefit from being open source thanks to improvements/github issues from the community. Our focus is making agents run easily and reliably in prod
The sales pressure from them on their agent core stuff has been really shocking over the last six months. Never seen anything like it.
So, basically:
1. Any tool result longer than 1500 tokens gets a link instead.
2. Forced compaction at 85%, removing all past messages apart from the last 4?
I'm sure these guys made it look good on the benchmarks, but the frontier labs have been hill-climbing this game for a while now, so odds are this just doesn't work for long-horizon tasks labs are optimizing for.
so far I have seen many harnesses that are built to provide similar quality to the provider's harness for lower cost. Has there been any projects that prioritizes quality and quality consistency?
is anyone using strands ? I looked into it and ended up picking mastra as it had a better story / seems more complete for people making agents in typescript.
"We noticed builders often wished their Claude Code or Codex setup could run in the cloud because locally their agent idea just “worked” with those harnesses.
But the moment you build your own agent, you’re on your own. It’s tricky wiring up the right primitives just well enough to match that “it just worked” feeling."
Im sorry, but who is saying this? If you just throw this statement into agent of your choice- and ask what native integrations exist to cover this use: OAI and Anthropic both have a handful of options here. Claude Agent SDK, Claude managed agents, Codex exec, Codex sdk, Codex app server, openai agents sdk, openai agents api.
Beyond that though, I'm certainly interested in the performance side of things. "Keep an eye out for a follow-up paper from our researchers regarding these benchmarks." Yes plz.
Pretty crazy amazon is advertising an open source repo. Not suggesting this is an ad but I've seen ads on reddit for it.
Ive heard from a 25 yoe consultant in a meetup group in person that aws agentcore was THE best way to handle enterprise agentic workflows with all of the proper knobs for governance etc since it comes with the iam integrations and arns etc.
I their web icon a pepe frog? hard pass if so..