Cascade Fusion and Correlation Enhancement for Knowledge Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) enhances compact student networks using knowledge from larger teacher networks.
• Existing KD methods face optimization challenges due to densely connected paths for multiscale feature transfer.
• Label correlations, crucial for intraclass similarity, are often overlooked in current KD approaches.

Purpose of the Study:

• To introduce Cascade Fusion and Correlation Enhancement for Knowledge Distillation (CC-KD).
• To simplify multiscale feature knowledge transfer and reduce optimization difficulty.
• To improve student network performance by incorporating label correlation into relational knowledge.

Main Methods:

• Implemented cascade fusion of multiscale features using cross-scale attention (CSA).
• Developed a method to enhance teacher logits by incorporating correlations among labels.
• Evaluated CC-KD on CIFAR100/10, ImageNet, RAF-DB, and FERPlus datasets.

Main Results:

• CC-KD significantly outperforms existing state-of-the-art methods.
• Achieved 71.70% accuracy on ImageNet and a new record of 90.20% on RAF-DB.
• Demonstrated superior performance with reduced computational cost and fewer parameters.

Conclusions:

• CC-KD effectively addresses optimization challenges in knowledge distillation.
• The proposed method enhances student model capabilities through simplified feature fusion and enhanced relational knowledge.
• CC-KD offers a more efficient and effective approach to knowledge distillation for deep learning models.