Multistage feature fusion knowledge distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Lightweight models typically exhibit lower recognition performance compared to large-scale models.
• Knowledge distillation offers a method to enhance lightweight model accuracy by leveraging knowledge from larger teacher models.
• Existing methods often struggle with effectively transferring implicit knowledge due to differing intermediate feature distributions.

Purpose of the Study:

• To develop an advanced knowledge distillation technique focusing on intermediate feature-level knowledge transfer.
• To improve the recognition accuracy of lightweight deep learning models.
• To address the challenge of learning from divergent feature distributions between teacher and student models.

Main Methods:

• Implemented a multistage feature fusion knowledge distillation method.
• Utilized a cross-stage feature fusion symmetric framework.
• Incorporated an attention mechanism for feature enhancement and a contrastive loss function for same-stage teacher-student alignment.

Main Results:

• Achieved superior performance compared to existing knowledge distillation methods.
• Boosted ResNet20 recognition accuracy on CIFAR100 from 69.06% to 71.34%.
• Increased ResNet18 recognition accuracy on TinyImagenet from 66.54% to 68.03%.

Conclusions:

• The proposed multistage feature fusion knowledge distillation method effectively improves lightweight model recognition accuracy.
• The approach demonstrates strong effectiveness and generalizability across different datasets and models.
• Further research is needed to optimize the distillation structure and feature extraction techniques.