Knowledge distillation is how a small, fast model learns to behave like a large, accurate one, keeping most of the quality while shedding the cost of running it. This category collects plain language breakdowns of the research that moves the field forward, from the classic KL divergence losses through newer token level and ranking based objectives. Every analysis explains the core idea, the math that makes it work, and the limitations the authors are honest about, and most ship with runnable code you can drop straight into your own training pipeline. It is built for practitioners who want to understand a method well enough to use it, not just cite it.
It is worth being upfront about what kind of paper this is. EasyDistill does not propose a new loss function or a new theoretical result about distillation. It is a systems paper, describing an open source toolkit that wraps a…
EasyDistill Packages LLM Distillation, But Its Top-k Shortcut Has A Cost Read More »
Streaming speech recognition built on Connectionist Temporal Classification, or CTC, has become a common choice because the architecture is simple and cheap to run in real time. Feed it audio in small chunks instead of the whole recording at once,…
Delayed-KD: Why Matching Every Frame Hurts Streaming Speech Recognition Read More »
Active learning exists to solve one specific pain point. Getting labels is expensive, so instead of labeling data at random, a model picks the unlabeled examples it expects to learn the most from, an expert labels just those, and the…
How ActiveKD Uses VLM Bias To Guide Active Learning Read More »
Cross modal knowledge distillation for RGBD data almost always follows the same recipe. Train a multimodal teacher that consumes both RGB and Depth, then train one or more single modality student models to mimic what that teacher learned, so the…
CroDiNo-KD: RGB and Depth Models That Train Each Other, No Teacher Needed Read More »
DeepSeek R1 made two things obvious to anyone paying attention. Reinforcement learning can pull genuinely new reasoning behavior out of a language model that supervised training alone never produces, and distilling from a strong reasoning model is often the faster,…
KDRL Unifies Distillation And Reinforcement Learning Read More »
Vehicle routing sounds like a solved problem until you actually try to solve it at scale. A logistics company does not have one routing problem. It has a family of them. Some trucks return to a central warehouse and some…
MTL-KD: Knowledge Distillation For Scalable Vehicle Routing Solvers Read More »
Large language models keep getting better, but that improvement comes bundled with a cost problem. A model with billions of parameters needs serious hardware just to run, which rules it out for a lot of practical deployment settings, particularly anything…
POCL: How A Progressive Overload Curriculum Improves LLM Distillation Read More »
Knowledge distillation has a well known shape. Train a large, accurate teacher model. Train a small student model to mimic the teacher’s softened output probabilities, using the temperature trick Geoffrey Hinton popularized, rather than only the ground truth labels. The…
Almost every logit based distillation method shares an awkward design. You take the student’s ordinary cross entropy loss, bolt a distillation term onto it, and then spend hours tuning the single weight that decides how much each half matters. Turn…
PLD: List Wise Knowledge Distillation with Plackett-Luce Read More »
Neural network potentials, NNPs for short, have become the go to tool for simulating how atoms move and interact without running full quantum mechanical calculations for every timestep. They learn the potential energy surface, essentially a map of how much…
Knowledge Distillation: Why an Untuned Teacher Model Trains Better MD Potentials Read More »