Tailored knowledge distillation with automated loss function learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge Distillation (KD) is a key technique for compressing large models.
• Current KD methods rely on manually designed, task-specific distillation losses.
• The efficacy of these handcrafted losses is often unclear.

Purpose of the Study:

• To introduce Learnable Knowledge Distillation (LKD), a novel approach for autonomous learning of distillation losses.
• To develop an adaptive, performance-driven distillation strategy.
• To enhance model compression without task-specific loss engineering.

Main Methods:

• Implemented a bi-level optimization and iterative strategy to learn distillation losses.
• Utilized generic loss networks for logits and intermediate features.
• Introduced dynamic optimization and uniform sampling of diverse student models for robust loss training.

Main Results:

• LKD demonstrated superior performance across various datasets (CIFAR, ImageNet) without task-specific adjustments.
• Achieved 73.62% accuracy on ImageNet using MobileNet, a 2.94% improvement over the KD baseline.
• Showcased enhanced adaptability and performance driven by learned, dynamic distillation losses.

Conclusions:

• LKD offers a universally adaptable distillation framework.
• Autonomous learning of distillation losses leads to significant performance gains.
• This method reduces the need for manual, task-specific loss design in model compression.