Streaming speech recognition built on Connectionist Temporal Classification, or CTC, has become a common choice because the architecture is simple and cheap to run in real time. Feed it audio in small chunks instead of the whole recording at once,…
Delayed-KD: Why Matching Every Frame Hurts Streaming Speech Recognition Read More »
Active learning exists to solve one specific pain point. Getting labels is expensive, so instead of labeling data at random, a model picks the unlabeled examples it expects to learn the most from, an expert labels just those, and the…
How ActiveKD Uses VLM Bias To Guide Active Learning Read More »
Cross modal knowledge distillation for RGBD data almost always follows the same recipe. Train a multimodal teacher that consumes both RGB and Depth, then train one or more single modality student models to mimic what that teacher learned, so the…
CroDiNo-KD: RGB and Depth Models That Train Each Other, No Teacher Needed Read More »
DeepSeek R1 made two things obvious to anyone paying attention. Reinforcement learning can pull genuinely new reasoning behavior out of a language model that supervised training alone never produces, and distilling from a strong reasoning model is often the faster,…
KDRL Unifies Distillation And Reinforcement Learning Read More »
Vehicle routing sounds like a solved problem until you actually try to solve it at scale. A logistics company does not have one routing problem. It has a family of them. Some trucks return to a central warehouse and some…
MTL-KD: Knowledge Distillation For Scalable Vehicle Routing Solvers Read More »
Large language models keep getting better, but that improvement comes bundled with a cost problem. A model with billions of parameters needs serious hardware just to run, which rules it out for a lot of practical deployment settings, particularly anything…
POCL: How A Progressive Overload Curriculum Improves LLM Distillation Read More »
Connectionist Temporal Classification (CTC) powers countless speech recognition systems. But here’s the dirty secret: its “context-independent” assumption is a myth. Modern encoders do learn context-dependent patterns, and ignoring this wastes potential. This paper reveals how to harness this hidden power,…
Knowledge distillation has a well known shape. Train a large, accurate teacher model. Train a small student model to mimic the teacher’s softened output probabilities, using the temperature trick Geoffrey Hinton popularized, rather than only the ground truth labels. The…
Almost every logit based distillation method shares an awkward design. You take the student’s ordinary cross entropy loss, bolt a distillation term onto it, and then spend hours tuning the single weight that decides how much each half matters. Turn…
PLD: List Wise Knowledge Distillation with Plackett-Luce Read More »
Neural network potentials, NNPs for short, have become the go to tool for simulating how atoms move and interact without running full quantum mechanical calculations for every timestep. They learn the potential energy surface, essentially a map of how much…
Knowledge Distillation: Why an Untuned Teacher Model Trains Better MD Potentials Read More »