cross-entropy

Almost every logit based distillation method shares an awkward design. You take the student’s ordinary cross entropy loss, bolt a distillation term onto it, and then spend hours tuning the single weight that decides how much each half matters. Turn…