What's this about?
On-policy distillation offers dense token-level supervision as an alternative to the sparse outcome-level advantages of RL with verifiable rewards. But the teacher scores student-generated trajectories that are inherently *off-policy for the teacher* — so how reliable that supervision is, and where the student's improvement actually comes from, is unclear. The paper quantifies it. 103 upvotes, the top paper of the day.
Why is the teacher scoring off-policy trajectories a problem?
Because the teacher is being asked to judge work it would never have produced. A strong model evaluating its own outputs is on familiar ground; evaluating a weaker model's strange intermediate steps is a different task, and its scores there are noisier than its capability suggests. If the supervision is noisy, then improvement attributed to "distilling the teacher's knowledge" may actually be the student self-improving with the teacher acting as a rough filter — which is a different mechanism with different scaling behaviour.
Why does this matter to you?
- On-policy distillation has been the technique of the month, with four papers in this digest's recent runs — this is the one asking whether it works the way everyone assumes.
- "From noisy teacher to self-improvement" in the subtitle is the finding: the mechanism may not be what the name says.
- It is the top-upvoted paper of the day, which suggests the field shares the doubt.