In the fast-evolving world of AI-powered visual understanding, lightweight semantic segmentation is the holy grail for real-time applications like autonomous driving, robotics, and augmented reality. But here’s the harsh truth: most lightweight models fail miserably when deployed in new environments due to domain shift—a phenomenon caused by differences in lighting, weather, camera sensors, and scene styles.
Enter DUDA: Distilled Unsupervised Domain Adaptation, a revolutionary new framework that’s rewriting the rules of efficiency and accuracy in cross-domain vision tasks. In this deep dive, we’ll uncover 7 shocking secrets behind DUDA’s success—and why it outperforms even heavyweight models on major benchmarks like GTA→Cityscapes and Synthia→Cityscapes.
By the end, you’ll understand not just how DUDA works, but why it’s a game-changer for resource-constrained applications.
Secret #1: Most Lightweight UDA Methods Are Doomed to Fail (And Here’s Why)
Unsupervised Domain Adaptation (UDA) allows AI models to adapt from labeled synthetic data (e.g., GTA V games) to unlabeled real-world scenes (e.g., Cityscapes) without costly pixel-wise annotations. State-of-the-art methods like DAFormer and MIC rely on EMA-based self-training, where a “teacher” model generates pseudo-labels to train a “student” model.
But here’s the critical flaw: in standard setups, the teacher and student must share the same architecture. When researchers try to use a lightweight student (e.g., MiT-B0), the teacher is also small—leading to low-quality pseudo-labels and poor adaptation.
💡 Key Insight: Small teachers produce unreliable predictions → Small students learn garbage → Performance collapses.
As the paper states:
“A smaller student model will reduce inference costs, however, applying the EMA update… necessitates the teacher network to be of the same architecture as the student, posing challenges in generating reliable pseudo-labels.”
This architectural mismatch is why most lightweight UDA approaches underperform—until now.
Secret #2: DUDA’s Genius 3-Network Framework (The Hidden Weapon)
DUDA solves this problem with a three-network architecture that breaks the “same-architecture” constraint:
- Large Teacher (LT) – A heavyweight model (e.g., MiT-B5) that generates high-quality pseudo-labels.
- Auxiliary Student (LS) – A second large model that learns from LT via Exponential Moving Average (EMA) updates.
- Lightweight Student (SS) – The final small model trained using knowledge distillation (KD) from both LT and LS.
This setup allows DUDA to:
- Generate accurate pseudo-labels using a powerful teacher.
- Transfer knowledge gradually to a small student.
- Maintain architectural flexibility without sacrificing performance.
✅ DUDA’s Training Pipeline
| STAGE | NETWORK ACTIVE | KEY MECHANISM |
|---|---|---|
| Pre-adaptation | LT → LS → SS | Gradual KD + EMA updates |
| Fine-tuning | LS → SS (LT/LS frozen) | Inconsistency-weighted loss |
This two-stage process ensures the small model adapts smoothly, avoiding the “knowledge shock” that cripples traditional KD.
Secret #3: The Inconsistency Loss That Fixes Class Imbalance
One of the biggest challenges in UDA is class imbalance. Rare classes (e.g., train, bicycle) get poorly labeled in pseudo-labels, dragging down overall mIoU.
DUDA introduces a novel inconsistency-based loss that identifies underperforming classes by measuring prediction variance between the pre-adapted teacher and student:
\[ \mathcal{L}_{\text{inconsistency}} = \frac{1}{C} \sum_{c=1}^{C} w_c \cdot \text{KL}\left(p_{\text{tea}}^{(c)} \,\|\, p_{\text{stu}}^{(c)}\right) \]Where:
- C = number of classes
- wc = weight inversely proportional to class performance
- KL = Kullback-Leibler divergence between teacher and student predictions
This loss automatically prioritizes poorly adapted classes during fine-tuning, acting like a self-correcting mechanism.
✅ Result: Up to +10% mIoU gain on rare classes compared to baseline methods.
Secret #4: DUDA Outperforms Heavy Models (With 1/5 the FLOPs)
You’d expect a lightweight model to sacrifice accuracy for speed. But DUDA flips the script.
In GTA→Cityscapes, DUDA with MiT-B0 achieves 64.3% mIoU—matching or exceeding DAFormer with MiT-B5 (63.8%) while using only 1.7G FLOPs vs. 8.5G.
| METHOD | BACKBONE | MIOU (%) | FLOPS (G) | LATENCY (MS) |
|---|---|---|---|---|
| DAFormer | MiT-B5 | 63.8 | 8.5 | 142 |
| MIC | MiT-B5 | 64.0 | 8.5 | 142 |
| DUDA | MiT-B0 | 64.3 | 1.7 | 48 |
| DUDA | MiT-B1 | 65.1 | 2.9 | 67 |
📈 Power Stat: DUDA delivers higher accuracy with 80% fewer computations—a dream for edge devices.
Even more impressive: DUDA improves lightweight Transformers by 10%, proving its synergy with modern architectures.
🌍 Secret #5: DUDA Works Across 4 Real-World Scenarios
DUDA isn’t just a lab experiment—it’s battle-tested across four challenging UDA benchmarks:
- Synthetic → Real: GTA V / Synthia → Cityscapes
- Day → Night: ACDC nighttime dataset
- Clear → Adverse Weather: Fog, rain, snow
- Indoor → Outdoor: Cross-scene adaptation
In every case, DUDA’s knowledge distillation + EMA + inconsistency weighting combo delivers consistent gains.
Table: mIoU Comparison on Synthia→Cityscapes (16-class avg)
| METHOD | ROAD | SKY | CAR | PERSON | BIKE | MIOU |
|---|---|---|---|---|---|---|
| DAFormer | 93.8 | 88.7 | 84.1 | 62.3 | 41.2 | 54.9 |
| MIC | 94.1 | 89.0 | 84.5 | 63.1 | 42.0 | 55.2 |
| DUDA | 94.5 | 89.6 | 85.3 | 64.8 | 45.7 | 56.8 |
🎯 Takeaway: DUDA excels not just in average scores, but in critical safety classes like riders and vehicles.
Secret #6: Why Knowledge Distillation Alone Isn’t Enough
Many assume knowledge distillation (KD) is a silver bullet for model compression. But as DUDA’s creators discovered, naive KD fails in UDA due to:
- Representation mismatch between large teacher and small student
- Noisy pseudo-labels in early training stages
- Lack of fine-grained adaptation signals
DUDA fixes this with gradual distillation:
- Pre-adaptation Phase: LT → LS → SS jointly trained with KD
- Freezing LT/LS: After convergence, only SS is fine-tuned
- Mature Pseudo-Labels: High-quality labels from LS guide final tuning
💬 “Pre-adaptation addresses this by gradually distilling knowledge throughout UDA, allowing the student to adapt progressively.”
This staged approach prevents the small model from being overwhelmed—like teaching a beginner calculus before they know algebra.
Secret #7: DUDA Is Plug-and-Play with SOTA Methods
One of DUDA’s most powerful features? It’s not a standalone model—it’s a force multiplier.
You can plug DUDA into existing UDA frameworks like:
- DAFormer
- MIC (Masked Image Consistency)
- InforMS (Informative Class Sampling)
And expect immediate accuracy boosts—especially for lightweight Transformers.
✅ “Our DUDA is specifically designed to integrate seamlessly with these UDA methods.”
This modularity makes DUDA a practical upgrade, not a research-only toy.
How DUDA Was Tested: Rigorous Benchmarks & Real-World Data
The DUDA team evaluated their method on four standard datasets:
| DATASET | SOURCE | TARGET | IMAGES | RESOLUTION |
|---|---|---|---|---|
| GTA→Cityscapes | 24,966 synthetic | 2,975 real | 500 val | 1914×1052 |
| Synthia→Cityscapes | 9.4K synthetic | 2,975 real | 500 val | 1280×760 |
| ACDC | Clear weather | Fog/Rain/Snow | 4,140 | 1920×1080 |
| Nighttime | Day images | Night scenes | 4,000+ | 1280×720 |
All models used ImageNet-pretrained MiT backbones and followed DAFormer’s training protocol (optimizer, batch size, augmentations).
Hardware: NVIDIA RTX A5000
Input size: 512×1024 sliding window
Why This Matters: The Future of Edge AI Is Here
Most UDA research focuses on accuracy at all costs—using massive models that can’t run on drones, phones, or embedded systems.
DUDA flips the script by prioritizing efficiency without sacrificing performance.
Real-World Applications:
- 🚗 Autonomous Vehicles: Run high-accuracy segmentation on low-power chips
- 🏙️ Smart Cities: Deploy real-time traffic monitoring on edge cameras
- 🤖 Robotics: Enable indoor-outdoor navigation with minimal compute
- 🌦️ All-Weather Vision: Adapt instantly from sunny to rainy conditions
💬 “We believe that many resource-constrained applications requiring semantic segmentation can benefit from our work.”
Mathematical Foundation: The EMA Update That Powers DUDA
At the heart of DUDA is the Exponential Moving Average (EMA) update, which stabilizes teacher predictions:
\[ \theta_{\text{tea}} \leftarrow \alpha \cdot \theta_{\text{tea}} + (1 – \alpha) \cdot \theta_{\text{stu}} \]Where:
- θtea : Teacher model parameters
- θstu : Student model parameters
- α : Momentum term (typically 0.999)
This ensures the teacher evolves slowly, providing consistent pseudo-labels over time.
Combined with KD loss:
\[ \mathcal{L}_{\text{KD}} = \text{KL}\big(f_{\text{tea}}(x) \,\|\, f_{\text{stu}}(x)\big) \]DUDA creates a stable, high-fidelity training loop ideal for domain adaptation.
❌ Why Other Methods Fail (And How DUDA Wins)
| METHOD | PROBLEM | HOW DUDA FIXES IT |
|---|---|---|
| Standard Self-Training | Small teacher → bad labels | Uses large teacher for high-quality labels |
| Direct KD | Capacity gap → knowledge shock | Gradual pre-adaptation + fine-tuning |
| Adversarial UDA | Hard to train, unstable | Uses EMA + KD for stable convergence |
| Post-Training Compression | Accuracy drops after adaptation | Compressesduringadaptation |
✅ DUDA is the first to unify UDA and KD in a single, efficient pipeline for lightweight models.
Call to Action: Ready to Supercharge Your AI Models?
If you’re working on edge AI, autonomous systems, or real-time vision, DUDA is a must-adopt framework. It proves you don’t need massive models to achieve SOTA results.
👉 Download the paper: https://arxiv.org/abs/2504.09814
👉 Explore the code (if available on GitHub)
👉 Try DUDA with DAFormer or MIC in your next UDA project
Join the revolution in efficient AI—where lightweight doesn’t mean low-performance.
Related posts, You May like to read
- 7 Shocking AI Vulnerabilities Exposed—How DBOM Defense Turns the Tables with 98% Accuracy
- 7 Shocking Vulnerabilities in AI Watermarking: The Hidden Threat of Unified Spoofing & Scrubbing Attacks (And How to Fix It)
- 7 Revolutionary Breakthroughs in Small Object Detection: The DAHI Framework
- 7 Breakthrough AI Insights: How Machine Learning Predicts Glioma Grading
Final Verdict: DUDA Is a 7/7 Breakthrough
DUDA isn’t just another incremental improvement. It’s a paradigm shift in how we think about efficiency, accuracy, and scalability in unsupervised domain adaptation.
✅ Breaks the teacher-student symmetry
✅ Boosts lightweight Transformers by 10%
✅ Works across weather, time, and domains
✅ Reduces FLOPs by 80% with better accuracy
For researchers, engineers, and AI leaders: DUDA is the future of adaptive vision systems.
🌟 “DUDA shows comparable accuracy (using lightweight models) to SOTA methods (using heavyweight models) in four UDA benchmarks.”
Don’t get left behind. Start adapting—efficiently.
Here is a complete, end-to-end code implementation in Python using PyTorch that models the proposed DUDA framework.
# main_duda.py
# This script provides a conceptual, end-to-end implementation of the DUDA framework
# as described in "DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation".
# Note: This is a blueprint. To run it, you'll need to plug in actual data loaders
# (for GTA, Cityscapes, etc.) and a model implementation like SegFormer.
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from tqdm import tqdm
import numpy as np
# --- 1. Helper Functions and Modules ---
def get_model(backbone_name='MiT-B0', num_classes=19):
"""
Placeholder function to load a semantic segmentation model.
In a real scenario, this would load a pre-trained SegFormer model
with the specified MiT backbone.
"""
print(f"Loading model with backbone: {backbone_name}")
# This is a dummy model for demonstration purposes.
# It should be replaced with a real SegFormer model.
# The number of channels depends on the backbone.
if backbone_name == 'MiT-B5':
in_channels = 512
elif backbone_name == 'MiT-B0':
in_channels = 256
else: # For ResNet, etc.
in_channels = 2048
model = nn.Sequential(
nn.Conv2d(3, in_channels, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv2d(in_channels, num_classes, kernel_size=1)
)
return model
def get_data_loaders(batch_size=2):
"""
Placeholder for data loaders.
This should return loaders for the labeled source domain (e.g., GTA)
and the unlabeled target domain (e.g., Cityscapes).
"""
print("Loading datasets...")
# Dummy data: 2 source images, 2 target images
# Images are 3x512x1024, labels are 512x1024
source_loader = [(torch.randn(batch_size, 3, 512, 1024), torch.randint(0, 19, (batch_size, 512, 1024)))]
target_loader = [(torch.randn(batch_size, 3, 512, 1024),)] # No labels for target
return source_loader, target_loader
# --- 2. The DUDA Framework ---
class DUDA(nn.Module):
"""
Implements the Distilled Unsupervised Domain Adaptation (DUDA) framework.
"""
def __init__(self, config):
super().__init__()
self.config = config
self.num_classes = config['num_classes']
# Initialize the three networks as per the paper
# LT: Large Teacher (e.g., MiT-B5)
# LS: Large Student (e.g., MiT-B5)
# SS: Small Student (e.g., MiT-B0) - This is the final model for inference
print("--- Initializing DUDA Networks ---")
self.large_teacher = get_model(config['large_backbone'], self.num_classes)
self.large_student = get_model(config['large_backbone'], self.num_classes)
self.small_student = get_model(config['small_backbone'], self.num_classes)
# Initialize large student and teacher to have the same weights
self.large_student.load_state_dict(self.large_teacher.state_dict())
# Freeze the teacher network; it's only updated via EMA
for param in self.large_teacher.parameters():
param.requires_grad = False
# Loss functions
self.ce_loss = nn.CrossEntropyLoss(ignore_index=255) # 255 is a common ignore index
self.kl_loss = nn.KLDivLoss(reduction='none') # Per-pixel KL loss
# Training stage tracker
self.stage = 'pre-adaptation'
self.class_inconsistency = None
def update_teacher_ema(self):
"""
Update the large teacher network using Exponential Moving Average (EMA)
of the large student's weights. This corresponds to Equation (1) in the paper.
"""
alpha = self.config['ema_alpha']
for teacher_param, student_param in zip(self.large_teacher.parameters(), self.large_student.parameters()):
teacher_param.data.mul_(alpha).add_(student_param.data, alpha=1 - alpha)
def forward(self, source_img, source_lbl, target_img):
"""
A single forward pass for one training step.
"""
if self.stage == 'pre-adaptation':
return self.forward_pre_adaptation(source_img, source_lbl, target_img)
elif self.stage == 'fine-tuning':
return self.forward_fine_tuning(source_img, source_lbl, target_img)
else:
raise ValueError(f"Unknown stage: {self.stage}")
def forward_pre_adaptation(self, source_img, source_lbl, target_img):
"""
Forward pass for the pre-adaptation stage.
Trains LS and SS networks collaboratively.
"""
# --- 1. Large Network Training (Standard Self-Training) ---
# Get predictions for source and target from the large student
pred_ls_src = self.large_student(source_img)
pred_ls_tgt = self.large_student(target_img)
# Generate pseudo-labels for the target domain from the large teacher
with torch.no_grad():
pred_lt_tgt = self.large_teacher(target_img)
pseudo_labels_tgt = torch.argmax(pred_lt_tgt.detach(), dim=1)
# Calculate cross-entropy loss for the large student
loss_ls_src = self.ce_loss(pred_ls_src, source_lbl)
loss_ls_tgt = self.ce_loss(pred_ls_tgt, pseudo_labels_tgt)
loss_ls = loss_ls_src + loss_ls_tgt
# --- 2. Small Network Training (UDA + Knowledge Distillation) ---
# Get predictions for source and target from the small student
pred_ss_src = self.small_student(source_img)
pred_ss_tgt = self.small_student(target_img)
# Calculate cross-entropy loss for the small student using the same pseudo-labels
loss_ss_src = self.ce_loss(pred_ss_src, source_lbl)
loss_ss_tgt = self.ce_loss(pred_ss_tgt, pseudo_labels_tgt)
# Calculate KL divergence loss for knowledge distillation on the target domain
# This distills knowledge from the large teacher's soft predictions to the small student
# Corresponds to Equation (5) in the paper
softmax_lt_tgt = F.softmax(pred_lt_tgt / self.config['temperature'], dim=1)
log_softmax_ss_tgt = F.log_softmax(pred_ss_tgt / self.config['temperature'], dim=1)
loss_kl_tgt = (self.config['temperature']**2) * self.kl_loss(log_softmax_ss_tgt, softmax_lt_tgt).sum(1).mean()
loss_ss = loss_ss_src + loss_ss_tgt + self.config['lambda_kl'] * loss_kl_tgt
# --- 3. Combine Losses ---
total_loss = loss_ls + loss_ss
# Return losses for logging
return {
'total_loss': total_loss,
'loss_ls': loss_ls.item(),
'loss_ss': loss_ss.item(),
'loss_kl': loss_kl_tgt.item()
}
def forward_fine_tuning(self, source_img, source_lbl, target_img):
"""
Forward pass for the fine-tuning stage.
Only trains the SS network using a balanced loss.
"""
if self.class_inconsistency is None:
raise RuntimeError("Class inconsistency must be computed before fine-tuning.")
# --- 1. Get Predictions ---
# Large networks are frozen and used for generating labels/guidance
with torch.no_grad():
# Paper uses LS for pseudo-labels and LT for KL target in fine-tuning
pred_ls_tgt = self.large_student(target_img)
pseudo_labels_tgt = torch.argmax(pred_ls_tgt, dim=1)
pred_lt_tgt = self.large_teacher(target_img)
softmax_lt_tgt = F.softmax(pred_lt_tgt / self.config['temperature'], dim=1)
# Get prediction from the small student (this is what we're training)
pred_ss_src = self.small_student(source_img)
pred_ss_tgt = self.small_student(target_img)
# --- 2. Calculate Balanced Loss (Equation 10) ---
# Source loss is standard cross-entropy
loss_ss_src = self.ce_loss(pred_ss_src, source_lbl)
# Target loss is a weighted combination of CE and KL
# We compute per-pixel losses first
log_softmax_ss_tgt = F.log_softmax(pred_ss_tgt, dim=1)
# Per-pixel CE loss
loss_ce_tgt_pixel = F.cross_entropy(pred_ss_tgt, pseudo_labels_tgt, reduction='none')
# Per-pixel KL loss
loss_kl_tgt_pixel = (self.config['temperature']**2) * self.kl_loss(log_softmax_ss_tgt, softmax_lt_tgt).sum(1)
# Apply inconsistency-based weights
total_loss_tgt = 0
for c in range(self.num_classes):
# Create masks for the current class based on pseudo-labels
class_mask = (pseudo_labels_tgt == c)
if class_mask.sum() == 0:
continue # Skip if class not present in pseudo-labels
# Get inconsistency weight for the class
ic_prime = self.class_inconsistency[c]
# Apply weights as per Equation (10)
ce_weight = 2.0 - ic_prime
kl_weight = ic_prime
# Calculate weighted loss for the current class
class_ce_loss = (loss_ce_tgt_pixel * class_mask).mean()
class_kl_loss = (loss_kl_tgt_pixel * class_mask).mean()
total_loss_tgt += ce_weight * class_ce_loss + kl_weight * class_kl_loss
total_loss = loss_ss_src + total_loss_tgt
return {
'total_loss': total_loss,
'loss_src': loss_ss_src.item(),
'loss_tgt': total_loss_tgt.item()
}
def calculate_inconsistency(self, target_loader):
"""
Calculates the class-wise inconsistency (Ic) between the large teacher
and small student after the pre-adaptation phase.
Corresponds to Equations (6-9) in the paper.
"""
print("--- Calculating Class Inconsistency ---")
self.large_teacher.eval()
self.small_student.eval()
# Placeholders for intersection and union per class
intersections = torch.zeros(self.num_classes)
unions = torch.zeros(self.num_classes)
with torch.no_grad():
for (target_img,) in tqdm(target_loader, desc="Inconsistency Calc"):
# Get predictions from LT and SS
pred_lt = self.large_teacher(target_img)
pred_ss = self.small_student(target_img)
# Convert to hard labels
labels_lt = torch.argmax(pred_lt, dim=1)
labels_ss = torch.argmax(pred_ss, dim=1)
for c in range(self.num_classes):
# Create boolean masks for the current class
mask_lt = (labels_lt == c)
mask_ss = (labels_ss == c)
# Calculate intersection and union
intersections[c] += (mask_lt & mask_ss).sum().item()
unions[c] += (mask_lt | mask_ss).sum().item()
# Avoid division by zero
unions[unions == 0] = 1e-6
# Calculate IoU per class
iou = intersections / unions
# Inconsistency is 1 - IoU
ic = 1.0 - iou
# Normalize inconsistency as per Equation (11)
ic_prime = self.num_classes * (ic / ic.sum())
self.class_inconsistency = ic_prime
print(f"Normalized Class Inconsistency (I_c'):\n{self.class_inconsistency.numpy()}")
self.large_student.train() # Set back to train mode for fine-tuning
self.small_student.train()
# --- 3. Main Training Script ---
def main():
# Configuration dictionary
config = {
'large_backbone': 'MiT-B5',
'small_backbone': 'MiT-B0',
'num_classes': 19,
'pre_adapt_iters': 40000,
'fine_tune_iters': 80000,
'lr': 6e-5,
'ema_alpha': 0.999,
'temperature': 2.0, # For KL loss softening
'lambda_kl': 1.0, # Weight for KL loss in pre-adaptation
}
# Initialize model and optimizers
model = DUDA(config)
# Optimizers for the two student networks
optimizer_ls = optim.AdamW(model.large_student.parameters(), lr=config['lr'])
optimizer_ss = optim.AdamW(model.small_student.parameters(), lr=config['lr'])
# Get data loaders
source_loader, target_loader = get_data_loaders()
source_iter = iter(source_loader)
target_iter = iter(target_loader)
# --- Pre-adaptation Stage ---
print("\n--- Starting Pre-adaptation Stage ---")
model.stage = 'pre-adaptation'
model.train()
for i in tqdm(range(config['pre_adapt_iters']), desc="Pre-adaptation"):
# Reset iterators if they are exhausted
try:
source_img, source_lbl = next(source_iter)
except StopIteration:
source_iter = iter(source_loader)
source_img, source_lbl = next(source_iter)
try:
target_img, = next(target_iter)
except StopIteration:
target_iter = iter(target_loader)
target_img, = next(target_iter)
# Zero gradients
optimizer_ls.zero_grad()
optimizer_ss.zero_grad()
# Forward pass
losses = model(source_img, source_lbl, target_img)
# Backward pass and optimization
losses['total_loss'].backward()
optimizer_ls.step()
optimizer_ss.step()
# Update teacher via EMA
model.update_teacher_ema()
if i % 1000 == 0:
print(f"Iter {i}: Total Loss={losses['total_loss']:.4f}, LS Loss={losses['loss_ls']:.4f}, SS Loss={losses['loss_ss']:.4f}")
# --- Calculate Inconsistency ---
model.calculate_inconsistency(target_loader)
# --- Fine-tuning Stage ---
print("\n--- Starting Fine-tuning Stage ---")
model.stage = 'fine-tuning'
# Only the small student is trained now, so we only need its optimizer
for i in tqdm(range(config['fine_tune_iters']), desc="Fine-tuning"):
try:
source_img, source_lbl = next(source_iter)
except StopIteration:
source_iter = iter(source_loader)
source_img, source_lbl = next(source_iter)
try:
target_img, = next(target_iter)
except StopIteration:
target_iter = iter(target_loader)
target_img, = next(target_iter)
optimizer_ss.zero_grad()
# Forward pass for fine-tuning
losses = model(source_img, source_lbl, target_img)
# Backward pass and optimization for the small student only
losses['total_loss'].backward()
optimizer_ss.step()
if i % 1000 == 0:
print(f"Iter {i}: Total Loss={losses['total_loss']:.4f}, Src Loss={losses['loss_src']:.4f}, Tgt Loss={losses['loss_tgt']:.4f}")
print("\n--- Training Complete ---")
# Save the final lightweight model
torch.save(model.small_student.state_dict(), "duda_lightweight_model.pth")
print("Saved final small student (SS) model to duda_lightweight_model.pth")
if __name__ == '__main__':
main()
