training techniqueIntroduced by Knowledge Distillation · 2015
Knowledge distillation
Train a small student to match a large teacher's softened predictions.
Drafted by AI · not yet reviewed
How this idea evolved
No earlier ideas recorded for this concept yet.
Papers using this
- 2019Noisy Student
- 2020GPT-3
- 2020DeiT
- 2021ALBEF
- 2022BEiT v2