A 1.5 billion parameter teacher knows things its 100 million parameter student will never quite learn. The question is how to transfer what can be transferred without warping the smaller model into something either too cautious or too greedy. The…
Alpha Beta Divergence Rebalances Knowledge Distillation Read More »
Knowledge distillation has a simple pitch. Train a big accurate model, then train a small cheap model to copy its answers rather than only the ground truth labels. The small model ends up smarter than it would from labels alone,…
FiGKD Distills Only the Teacher’s High Frequency Logits Read More »
Almost every logit based distillation method shares an awkward design. You take the student’s ordinary cross entropy loss, bolt a distillation term onto it, and then spend hours tuning the single weight that decides how much each half matters. Turn…
PLD: List Wise Knowledge Distillation with Plackett-Luce Read More »