Highlight Every Step: Knowledge Distillation via Collaborative Teaching.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks face deployment challenges on resource-constrained devices due to high costs.
• Knowledge distillation (KD) trains smaller networks using knowledge from larger ones.
• Existing KD methods often overlook valuable training process information.

Purpose of the Study:

• To develop an advanced KD strategy for efficient model compression.
• To improve the performance of compact student networks.

Main Methods:

• Introduced Collaborative Teaching KD (CTKD) with two distinct teachers.
• A 'scratch teacher' provides step-by-step guidance using temporary outputs.
• An 'expert teacher' directs the student to critical, task-relevant information.

Main Results:

• CTKD significantly boosts student network performance.
• The method achieves state-of-the-art results across multiple benchmark datasets (CIFAR-10/100, SVHN, Tiny ImageNet, ImageNet).
• Experimental validation confirms the efficiency and effectiveness of the proposed KD approach.

Conclusions:

• The CTKD strategy effectively leverages dual-teacher knowledge for superior model compression.
• This approach offers a promising solution for deploying deep learning models on devices with limited resources.