What's this about?
Nine harnesses across twelve configurations, tested on identical software engineering tasks with the same model (Kimi K3) and the same runtime — 360 trials, every one starting from the same fresh checkpoint restore, on identical vCPU, memory, disk and memory state. Pass rates run from 50.0% to 66.7%. Cost per run runs from $1.05 to $18.34. Codex leads on quality at 66.7% for $3.47; Claude Code ties for second on quality at 63.3% but costs $18.34; Exo Harness gets 53.3% for $1.05.
What is a "harness" and why would it change cost 17×?
The harness is everything wrapped around the model: the loop that calls it, the tools it can use, how much context it re-sends each turn, whether it re-reads files, how it retries. Cost is tokens, and tokens are almost entirely a function of those decisions rather than of the model. A harness that re-sends the whole repository every step costs an order of magnitude more than one that keeps a tight working set, for the same task and the same underlying intelligence. Quality varies far less than cost, which is the finding: you are mostly not paying for correctness.
What's the community saying?
- The line that summarises it: "Software is constrained when you write it. Agents have to be constrained while they run."
- Speed gets attention too — DeepSeek is consistently fastest, Kimi slowest, GPT in the middle, with Haiku noted as extremely fast when it fits the task.
- A practical objection: nobody is going to run 17 benchmark passes on a $50–100 task. The counter is that large organisations will, once benchmarking cost is dwarfed by org-wide savings.
- Multiple people immediately asked how to submit their own harness, which is the right response to a benchmark like this.