What's this about?
AutoSciBench watches how agents solve scientific tasks, then revises concepts and construction recipes to close shortcuts and expose new failure modes.
What exactly is benchmark saturation?
Once models score near the ceiling—or learn quirks of a fixed test—the benchmark stops distinguishing genuine capability improvements.
Why does this matter?
Adaptive tasks were harder yet rated higher quality across three domains. Automatic judges and generators can also introduce their own biases, so human domain oversight remains necessary.