What's this about?
When distilling from several teacher models across several domains, which teacher should the student learn from, on which domain, and when? D³-MOPD schedules that adaptively rather than fixing the mixture in advance. 22 upvotes.
What is distillation?
Training a small model to imitate a large one. The small model learns from the big model's outputs rather than from raw data, which transfers a surprising amount of capability at a fraction of the size. Multi-teacher distillation uses several large models, each strong in different areas — and then the problem becomes deciding who to listen to about what, which is exactly what a fixed mixture gets wrong.
Why does this matter to you?
- Distillation is the machinery behind the small models today's top HN thread is about.
- Two multi-teacher distillation papers trend on the same day (see Open-MOPD), which suggests an active competitive front.
- Adaptive scheduling is the same idea as curriculum learning, applied to teachers rather than examples.