What's this about?
A methodological critique of automatic harness evolution — systems that automatically improve an agent's scaffolding. Current practice searches harness configurations using a public benchmark's unit tests, then reports final performance on the *same* benchmark; the paper argues this is overfitting dressed as progress and proposes a cleaner protocol (HF 6↑).
What exactly is a "harness"?
The harness is everything around the model that turns it into a working agent: system prompts, tools, retry loops, memory management. Since harness design hugely affects agent performance, researchers built systems that evolve harnesses automatically — but if you tune the harness on the test you'll be graded on, you've memorized the exam, not gotten smarter.
Why does it matter?
Agent leaderboard scores increasingly reflect harness tuning rather than general capability; this is the kind of eval-hygiene work that decides whether reported agent progress is real.