What's this about?
Learning process rewards by identifying where a trajectory first went wrong, rather than scoring the whole thing. 9 upvotes.
What is a process reward, and why the first mistake?
An outcome reward scores the final answer; a process reward scores the steps. Process rewards are much more informative and much harder to obtain, because you need to know which step was bad. The insight here is that in a failed trajectory, everything after the first mistake is contaminated — the model was reasoning correctly about a situation that had already gone wrong. So the only step that carries a clean signal is the first one that broke, and locating it turns a long useless trajectory into one precise training example.
Why does this matter to you?
- It is the same observation as the Handoff Tax and DART-SD: recorded trajectories contain far less usable signal than their length suggests.
- "Everything after the first error is contaminated" is a good heuristic for reading agent logs as a human, too.
- It is a cheaper route to process supervision, which is otherwise the expensive part of training reasoning models.