What's this about?
AI has assisted research for a long time, but LLMs plus agentic scaffolds have changed the shape of it — a single system can now carry a project from initial hypothesis through to a finished paper. This work (23 upvotes) runs an end-to-end diagnostic evaluation across 100 real-world frontier research tasks, focused on where and how agents break rather than on a success rate.
What exactly is a "diagnostic" evaluation?
A standard benchmark returns a score: the agent completed 34% of tasks. That tells you nothing actionable, because a system that fails at the first step and one that does everything right but botches the final write-up produce the same number. A diagnostic evaluation instruments the whole pipeline and reports where things went wrong — hypothesis formation, experiment design, execution, interpretation, writing. It's the difference between a test result and a fault report, and it's what you need if the goal is fixing the system rather than ranking it.
Why is this worth noticing?
- Research agents are being marketed on their end-to-end capability, and end-to-end is exactly where compounding errors hide — an 80% success rate at each of five stages is a 33% success rate overall.
- Using 100 *real* frontier research tasks rather than constructed ones addresses the contamination and realism problems that make most research-capability claims hard to evaluate.
- It fits this week's broader shift toward benchmarking the research process itself — alongside Apodex, the replication work, and the Reconstruction benchmark.