What's this about?
The day's runaway top paper (151 upvotes) has an elegantly simple idea. On-policy distillation normally depends on an *informative asymmetry* between teacher and student — a bigger teacher, or privileged information like reference answers or ground-truth regions of interest. The authors ask where asymmetry can come from when nothing privileged is available, and answer by inverting it: rather than adding information to the teacher, subtract it from the student. Show the student a strongly augmented view of the image while the teacher sees the original, and you get the same effective learning signal for free — no annotations, no rewards, no separate stronger model.
What exactly is distillation, and what does "on-policy" add?
Distillation trains a small model to imitate a larger one — the student learns from the teacher's outputs rather than from raw labels, which transfers not just answers but the teacher's uncertainty and reasoning. Ordinary distillation trains on a fixed dataset. On-policy means the student generates its own attempts and the teacher corrects those specific attempts, so training targets exactly where the student is going wrong. The catch has always been that you need a teacher who genuinely knows more — which is what this paper removes by handicapping the student instead.
Why is this worth noticing?
- It removes the main cost of the technique. No labels, no reward model, no larger teacher — the asymmetry is manufactured for free from image augmentation.
- "Subtract from the student" is a genuinely reusable idea. The same inversion could apply anywhere distillation is bottlenecked by needing a stronger teacher.
- It lands beside SimpleOPD in the same batch, which attacks a different on-policy distillation obstacle (tokenizer mismatch between long-context teachers and short-context students) — a sign this technique is where a lot of current effort is going.