Distilling a Powerful Student Model via Online Knowledge Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing online knowledge distillation methods have limitations, such as ignoring information from some students or increasing computational complexity with ensemble models.
• These limitations hinder efficient and effective knowledge transfer in deep learning models.

Purpose of the Study:

• To propose a novel online knowledge distillation method, Feature Fusion and Self-Distillation (FFSD), that addresses the drawbacks of existing approaches.
• To improve the performance of a leader student model by leveraging information from multiple common student models without increasing deployment costs.

Main Methods:

• FFSD splits student models into leader and common sets, fusing features from common students to guide the leader student's learning.
• An enhancement strategy increases diversity among students, while a self-distillation module promotes generalization by mimicking deeper layer features in shallower ones.
• The method utilizes a feature fusion module and a self-distillation module within a unified framework.

Main Results:

• The proposed FFSD method achieves superior performance compared to existing online knowledge distillation techniques.
• Experiments on CIFAR-100 and ImageNet datasets validate the effectiveness of FFSD.
• The leader student, trained using FFSD, demonstrates enhanced performance without additional storage or inference costs.

Conclusions:

• FFSD offers an effective and efficient solution for online knowledge distillation by intelligently utilizing information from multiple student models.
• The method provides a unified framework that improves model performance and generalization capabilities.
• FFSD represents a significant advancement in knowledge distillation, offering practical benefits for deep learning deployments.