Leveraging logit uncertainty for better knowledge distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation aims to transfer knowledge from a larger teacher model to a smaller student model.
• Larger teacher models do not always yield better distillation due to architecture and output discrepancies.
• Teacher model confidence in predictions is crucial for effective knowledge transfer.

Purpose of the Study:

• To propose a novel knowledge distillation method that addresses the limitations of using large teacher models.
• To improve the performance of student models by effectively transferring knowledge from teacher models.
• To bridge the gap between teacher and student models with differing architectures.

Main Methods:

• Logits Uncertainty Distillation (LUD) is proposed, incorporating category uncertainty weighting.
• A confidence threshold and mask are used to discount uncertain classes during distillation.
• Two Spearman correlation loss functions align teacher and student model logits at category and sample levels.
• Adaptive dynamic temperature factors optimize the distillation process.

Main Results:

• The proposed LUD method enhances knowledge distillation effectiveness.
• Effective knowledge transfer is achieved even with significant architectural differences between models.
• Experiments across multiple datasets validate the method's performance.

Conclusions:

• Logits Uncertainty Distillation (LUD) offers a robust approach to improve student model performance.
• The method effectively handles uncertainty in teacher predictions for better knowledge transfer.
• LUD facilitates more efficient and accurate knowledge distillation, particularly in heterogeneous model settings.