What's this about?
On-policy distillation is a promising way to transfer reasoning from stronger teachers, but applying it with a long-context teacher and a short-context student creates practical obstacles — tokenizer mismatch among them. SimpleOPD (27 upvotes) is a simple method that works across tokenizers.
What exactly is a tokenizer mismatch?
Models don't read characters; they read tokens — chunks of text produced by a tokenizer, and different model families chop text differently. The same sentence might be 18 tokens for one model and 22 for another, split at different points. Distillation works by matching the student's probability distribution over next tokens to the teacher's — but if the two use different vocabularies, those distributions aren't over the same things and can't be compared directly. It's a mundane engineering obstacle that quietly restricts distillation to models from the same family.
Why is this worth noticing?
- Removing the same-family constraint is the practical win: you could distil from whichever teacher is actually best rather than whichever shares your tokenizer.
- The long-context-teacher, short-context-student direction matters economically, since it's how you get expensive extended reasoning into a model cheap enough to serve.