What's this about?
Diagnosing and fixing capability imbalance in multi-teacher on-policy distillation — where a student ends up strong in the areas its dominant teacher was strong in, and weak everywhere else. 17 upvotes.
What is "capability imbalance"?
If one teacher contributes most of the training signal, the student inherits that teacher's shape: its strengths, its blind spots, its style. You wanted the union of several teachers' abilities and you got a slightly worse copy of one. Diagnosing it requires measuring per-capability rather than in aggregate, because an averaged benchmark score hides exactly this failure.
Why does this matter to you?
- Paired with D³-MOPD above, this is the diagnosis to that paper's scheduling fix.
- The failure mode generalises: any ensemble where one source dominates produces the same shape.
- "On-policy" distillation was also the mechanism behind yesterday's SecOPD prompt-injection defence — the technique is showing up everywhere.