- Mines tasks from your merged PRs/commits
- Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc)
- Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
Thank you! Both of those papers are super new and super relevant.
The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.
PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.
Thanks for sharing this. For session-shaped benchmarks, how would you keep the evaluation fair when cache state and accumulated context differ across Claude Code and Codex runs?
Here's my current plan, the "session" will be made up of multiple SWE bench tasks stitched together.
Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs).
The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.
I've actually been working on a solution for this problem! https://www.stet.sh/
At a high level, it
- Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that just came out on this https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.30573
Thank you! Both of those papers are super new and super relevant.
The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.
PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.
Looking for feedback and thoughts. Here's a link to my one-page spec: https://docs.google.com/document/d/e/2PACX-1vRu5Fv5-KTJDnCEx...
Thanks for sharing this. For session-shaped benchmarks, how would you keep the evaluation fair when cache state and accumulated context differ across Claude Code and Codex runs?
Here's my current plan, the "session" will be made up of multiple SWE bench tasks stitched together.
Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs).
The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.