logit swapping

Every paper on knowledge distillation since 2015 has assumed something most practitioners would rather not say out loud. The teacher is right. The student should match it. If the student starts diverging from the teacher’s confidence distribution, the loss function…