Imagine watching someone take an expert’s detailed reasoning, strip out everything except the most important cues, and hand those cues to a student who has never seen the full picture. That is roughly what a team from National Cheng Kung…
Every paper on knowledge distillation since 2015 has assumed something most practitioners would rather not say out loud. The teacher is right. The student should match it. If the student starts diverging from the teacher’s confidence distribution, the loss function…
How Swapped Logit Distillation Fixes Wrong Teachers Read More »
A 1.5 billion parameter teacher knows things its 100 million parameter student will never quite learn. The question is how to transfer what can be transferred without warping the smaller model into something either too cautious or too greedy. The…
Alpha Beta Divergence Rebalances Knowledge Distillation Read More »