What's this about?
A benchmark led by Stanford researchers and built by the Terminal-Bench team, in collaboration with domain experts across scientific fields. The first release has 70 tasks drawn from the life, physical, Earth, mathematical and engineering sciences — workflows from researchers' own work, not tasks invented by model developers or data vendors. The strongest model evaluated, Claude Opus 5, resolves 30%.
Why does "who writes the benchmark" matter?
Because benchmarks decide what gets optimised. If model developers write the tasks, the tasks drift toward what models are close to solving, and scores rise without capability necessarily rising. If working scientists write them from their actual pipelines, the tasks include the parts nobody would think to include — the file formats, the half-documented tools, the steps that only make sense if you know the field. That is also why a 30% score is more informative than a 90% one on an easier set.
What's the community saying?
- Broad agreement on the premise: evaluating real research workflows is the right direction, and most agent benchmarks are toy tasks.
- The result people find surprising is Opus 5 outperforming Fable, when several report Fable feeling better for coding — with the suggested explanation that this measures science, not code.
- A concrete anecdote in support: Fable's safety behaviour once refused to work on a bacterial pathogenicity-island analysis pipeline, which is a real obstacle in biomedical work.
- One complaint: no Gemini in the evaluation.