Knowledge distillation is how a small, fast model learns to behave like a large, accurate one, keeping most of the quality while shedding the cost of running it. This category collects plain language breakdowns of the research that moves the field forward, from the classic KL divergence losses through newer token level and ranking based objectives. Every analysis explains the core idea, the math that makes it work, and the limitations the authors are honest about, and most ship with runnable code you can drop straight into your own training pipeline. It is built for practitioners who want to understand a method well enough to use it, not just cite it.
An engineer has a network that scores well on the validation set and a latency budget it misses by a factor of ten. She trains a model a tenth the size on the same labels and loses accuracy she cannot…
Knowledge Distillation Mathematics Explained in Depth Read More »
The idea underneath this work sounds strange the first time you hear it. Instead of saving a measurement as the array of numbers it obviously is, you train a neural network to output the value at any coordinate you ask…
WIEN-INR Compresses Scientific Data With Wavelets Read More »
Anyone who has spent time pruning networks has probably run into this pattern. You strip away a chunk of weights before training even starts, retrain the smaller network, and the test accuracy comes back close to the original, sometimes a…
Why Pruning a Neural Network Before Training Can Improve Its Generalization Read More »
Researchers at UT Arlington built a data-free knowledge distillation framework for tabular models that borrows a principle from software testing — systematically covering all pairs of feature interactions — and achieves the best student-teacher agreement in 14 of 16 benchmark…
TabKD: Data-Free Knowledge Distillation for Tabular Models via Interaction Diversity Read More »
A novel PUF-based framework embeds unclonable hardware signatures into teacher models during knowledge distillation, enabling post-theft traceability without architectural changes or runtime overhead.
PDF: PUF-based DNN Fingerprinting for Knowledge Distillation Traceability Read More »
Forecasting research has spent the last few years arguing with itself. Transformers arrived promising to capture long range dependencies across a time series, and models like Informer and Autoformer chased that promise with increasingly elaborate attention mechanisms. Then a 2023…
How TimeDistill Teaches a Lightweight MLP to Outperform Transformer Forecasters Read More »
MTKD-RL from the Institute of Computing Technology, Chinese Academy of Sciences — an RL agent arbitrates teacher weights dynamically, improving student accuracy across image classification, object detection, and semantic segmentation.
Multi-Teacher Knowledge Distillation with RL — Teaching the Agent Which Teacher to Trust Read More »
Convolutional networks are good at spotting tumors, cataracts and the subtle atrophy patterns that show up in Alzheimer scans. They have been good at this for years. What they are not good at, on their own, is telling a physician…
Knowledge Distillation Helps Medical AI Explain Its Choices Read More »
For about five years, relation based knowledge distillation has been the polite loser of the field. Instance matching methods kept winning benchmarks, while the relation based literature went quiet. A paper from Shanghai Jiao Tong University and the University of…
How Virtual Relations Revive Knowledge Distillation Read More »
Most feature based knowledge distillation work follows a familiar script. Take a bulky teacher network, decide which regions of its feature maps look most informative, then push the smaller student network to mimic exactly those regions. FGFI drew boxes near…
ACAM-KD Gives Student Networks A Say In Their Own Distillation Read More »