Knowledge distillation based on multi-layer fusion features.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation trains smaller student networks using knowledge from larger, pre-trained teacher networks.
• Current methods often use simple features from limited teacher network layers, neglecting richer, fused information.

Purpose of the Study:

• To propose Multi-feature Fusion Knowledge Distillation (MFKD) for improved knowledge transfer.
• To leverage expressive, fused features from diverse teacher network layers.

Main Methods:

• Extracting feature maps from multiple network layers (front, middle, bottom) of the teacher network.
• Designing a multi-feature fusion scheme to integrate these diverse features.
• Training student networks using the enriched, fused features.

Main Results:

• The fused feature map contains more meaningful information than single-location features.
• MFKD improved Top-1 accuracy by 1.82% for ResNet20 and 3.35% for VGG8 on CIFAR-100.
• Outperformed several state-of-the-art knowledge distillation methods.

Conclusions:

• MFKD enables student networks to learn effectively from richer, fused knowledge.
• The proposed method offers a superior approach to knowledge distillation for enhanced model performance.