What's this about?
EarlyEval (111 upvotes) cuts agent-evaluation cost by halting a run the moment its outcome can be predicted — because a single frontier-model pass over an agentic benchmark costs hundreds to thousands of dollars, paid again for every experiment.
What exactly makes evaluation so expensive?
Agent benchmarks run multi-step episodes — tool calls, retries, long contexts — so one benchmark pass is thousands of model calls. Evaluation spend can rival training spend; ARC's Astra runs cost tens of thousands per configuration.
Why is this worth noting?
Cheap evaluation is a research-velocity multiplier: teams iterate as fast as they can measure. An early-exit predictor is the kind of unglamorous infrastructure that quietly speeds up everyone.