What's this about?
"Harness" here means the surrounding system — evolutionary search loops, retry strategies, tool orchestration — that turns a raw model into an effective automated-discovery agent (e.g., for scientific hypothesis search or code optimization). This paper systematically decomposes two popular harness styles (OpenEvolve-style evolutionary search and TTT-Discover) and tests 30 budget-matched configurations across 12 model-problem pairs, using over 3.1 million LLM rollouts.
What exactly does "budget-matched" mean here?
It means every harness configuration being compared got the same total compute/API budget, so differences in results reflect harness design rather than one system simply being given more resources — a rigor bar that's easy to skip and hard to trust results without.
Why does it matter?
The finding that no single harness reliably wins, and that adaptive allocation across multiple harnesses beats committing to one, pushes against a common assumption in agent-tooling circles that there's a single "best practice" scaffold to copy. It reframes harness selection itself as a tunable parameter of the system, not a one-time architectural decision.