Lightweight Deep Learning

Anchor-Based Knowledge Distillation (AKD), a breakthrough in trustworthy AI for efficient model compression.

Anchor-Based Knowledge Distillation: A Trustworthy AI Approach for Efficient Model Compression

3 Comments / Machine Learning /

In the rapidly evolving field of artificial intelligence (AI), knowledge distillation (KD) has emerged as a cornerstone technique for compressing powerful, resource-intensive neural networks into smaller, more efficient models suitable for deployment on mobile and edge devices. However, traditional KD…

Anchor-Based Knowledge Distillation: A Trustworthy AI Approach for Efficient Model Compression Read More »

Inside LSSKD, a Self Supervised Distillation Framework That Trains Small Models Without a Teacher

Inside LSSKD, a Self Supervised Distillation Framework That Trains Small Models Without a Teacher

1 Comment / Machine Learning, Knowledge Distillation /

Knowledge distillation has a well known shape. Train a large, accurate teacher model. Train a small student model to mimic the teacher’s softened output probabilities, using the temperature trick Geoffrey Hinton popularized, rather than only the ground truth labels. The…

Inside LSSKD, a Self Supervised Distillation Framework That Trains Small Models Without a Teacher Read More »