Dual-Forward Path Teacher Knowledge Distillation (DFPT-KD)

The idea of knowledge distillation goes back to Hinton, Vinyals, and Dean, who proposed training a small student network to mimic the softened output probabilities of a large teacher rather than only the hard labels. It works remarkably well in…