Harness design cuts agent cost and failure rates, from Strands to GPT-6 Luna

What changed · following since Oct 14 Oct 25 — A controlled study finds harness swaps barely move coding-agent pass rates beyond noise, but can triple costs.

Early reports that harness design drives reliability and efficiency now face a direct challenge: a controlled study finds swapping harnesses barely moves pass rates beyond statistical noise, while costs can triple. The divergence may stem from methodology—earlier benchmarks often relied on single runs, while the new work holds models constant across multiple runs to isolate harness effects. The net thesis is now contested: harness choice may matter less for accuracy than for token efficiency, and some designs actively inflate cost without measurable gain.

• Strands compared six harnesses across eight benchmarks and 12 models in roughly 94,000 trials, finding about 34% lower token usage for its harness while maintaining accuracy with Claude, GPT and open-weight models.
• In Martin Fowler's coding-agent experiment, GPT-6 Luna succeeded in 3 of 10 unharnessed runs but all 10 harnessed runs; the harnessed runs cost $0.273 each versus $0.406 for GPT-6 Sol, a 33% lower cost.
• LangChain reported that changing only the harness moved its coding agent from the Top 30 to the Top 5 on Terminal Bench 2.0, using self-verification, tracing and context optimization.
• A Shandong University study found switching between harnesses like Claude Code, OpenCode, or mini-SWE-agent barely moved pass rates more than rerunning the same setup, but cost per task could triple.
• The Shandong researchers attribute earlier 20-point gaps between harnesses to single-run measurements; pure reruns on SWE-bench Verified already vary by up to six points.

Bottom line The harness question is now genuinely open: benchmark it yourself on your own tasks, with multiple runs, before assuming a new harness will help or hurt.