Decoupled Classifier Knowledge Distillation

Area of Science:

• Machine Learning
• Computer Vision

Background:

• Knowledge distillation methods like self-distillation, offline, online, output-based, and feature-based distillation are typically used independently.
• Combining existing distillation methods often leads to redundant information, computational waste, and increased complexity.

Purpose of the Study:

• To explore a novel approach for integrating distillation methods that aligns complex features without conflicting with output alignment.
• To propose a compromise solution that enhances the effectiveness of combining different distillation strategies.

Main Methods:

• Decoupling the classifier's output into non-target classes (student-learned) and target classes (teacher- and student-learned).
• Introducing Decoupled Classifier Knowledge Distillation (DCKD), which fixes acquired knowledge and encourages output alignment with the teacher model.
• Integrating relational-based and feature-based distillation for improved efficiency and flexibility.

Main Results:

• DCKD achieves superior results on CIFAR-100 and ImageNet datasets for image classification and object detection compared to single distillation methods.
• The proposed method enhances training efficiency without reduction.
• DCKD enables more efficient and flexible operation of relational-based and feature-based distillation.

Conclusions:

• DCKD demonstrates significant potential in integrating diverse knowledge distillation methods.
• This approach offers a promising direction for future research in distillation techniques.
• The method effectively merges distillation strategies, improving model performance and efficiency.