How Swapped Logit Distillation Fixes Wrong Teachers
Analysis by the aitrendblend editorial team · Pillar 2, Knowledge Distillation · Reading time about 12 minutes knowledge distillation swapped logit distillation SLD logit processing pseudo teacher loss scheduling CIFAR-100 ImageNet The standard distillation recipe trusts the teacher even when the teacher is wrong. SLD swaps the misclassified target back into the top slot before […]
How Swapped Logit Distillation Fixes Wrong Teachers Read More »