What's this about?
Taste-Bench mines moments where an engineering or research trajectory could take several directions, then asks a model to choose without seeing future outcomes.
What does “taste” mean here?
It means selecting a promising hypothesis or implementation before the evidence is complete. The benchmark reduces that judgement to multiple-choice forks, which is measurable but narrower than human taste.
Why does this matter?
author-reported preprint; 113 HF upvotes.
- The best tested model answers only 59.7% correctly.
- More reasoning tokens do not fix forks whose evidence appears later.
- Distilling an outcome-aware teacher improves held-out task success.