Knowledge distillation is how a small, fast model learns to behave like a large, accurate one, keeping most of the quality while shedding the cost of running it. This category collects plain language breakdowns of the research that moves the field forward, from the classic KL divergence losses through newer token level and ranking based objectives. Every analysis explains the core idea, the math that makes it work, and the limitations the authors are honest about, and most ship with runnable code you can drop straight into your own training pipeline. It is built for practitioners who want to understand a method well enough to use it, not just cite it.
For about five years, relation based knowledge distillation has been the polite loser of the field. Instance matching methods kept winning benchmarks, while the relation based literature went quiet. A paper from Shanghai Jiao Tong University and the University of…
How Virtual Relations Revive Knowledge Distillation Read More »
Most feature based knowledge distillation work follows a familiar script. Take a bulky teacher network, decide which regions of its feature maps look most informative, then push the smaller student network to mimic exactly those regions. FGFI drew boxes near…
ACAM-KD Gives Student Networks A Say In Their Own Distillation Read More »
Imagine watching someone take an expert’s detailed reasoning, strip out everything except the most important cues, and hand those cues to a student who has never seen the full picture. That is roughly what a team from National Cheng Kung…
Think about what a keypoint detector actually does. It takes an image, runs it through a large backbone, and returns a set of 2D coordinates corresponding to corners or landmark points on a 3D object. Those coordinates feed into a…
Uncertainty-Aware Knowledge Distillation for 6DoF Pose Estimation Read More »
Knowledge distillation for segmentation usually assumes the teacher and the student came from the same architectural gene pool. Train a big ResNet, shrink it into a small ResNet, transfer the soft labels or the intermediate features, done. That assumption made…
HeteroAKD Bridges CNN and Transformer Segmentation Models Read More »
Every knowledge distillation paper treats the training set as a fixed resource to be mined equally, epoch after epoch. A team from Zhejiang University spent a preprint asking whether that assumption was actually costing accuracy rather than protecting it, and…
Smarter Sample Selection for Video Model Compression with SAKD Read More »
Generative watermarking has become the default answer to a question every LLM provider eventually faces. If someone copies my model’s outputs, how do I prove it. Methods like KGW nudge token probabilities toward a hidden green list of words, so…
Knowledge Distillation Can Forge and Erase LLM Watermarks Read More »
Anyone who has worked on deep learning in healthcare quickly discovers that the academic benchmark setting — one big dataset, one train-test split, one model — rarely survives contact with the real world. Hospital systems accumulate data incrementally. Different sites…
Every paper on knowledge distillation since 2015 has assumed something most practitioners would rather not say out loud. The teacher is right. The student should match it. If the student starts diverging from the teacher’s confidence distribution, the loss function…
How Swapped Logit Distillation Fixes Wrong Teachers Read More »
Gleason grading, the system pathologists use to score prostate cancer aggressiveness from tissue samples, has a documented interobserver variability of about 40 percent among trained pathologists looking at the same slide. That number alone tells you the task is genuinely…
Uncertainty Aware Knowledge Distillation for Imbalanced Disease Grading Read More »