What's this about?
The study finds teachers and students can assign stopping probability to different EOS tokens. Distillation can then suppress the student's preferred stop without teaching a reliable replacement.
What exactly is an EOS token?
End-of-sequence is the model's learned action for stopping generation. Several token IDs can be functionally equivalent to a decoder but not equivalent in the training loss.
Why does this matter?
Author-reported preprint; 94 HF upvotes at fetch time.
- Excess length wastes inference cost and harms usability.
- Aligning decoder settings alone does not repair the training signal.
- The proposed fix reduces only one of several causes of rambling.