Attention and feature transfer based knowledge distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing knowledge distillation (KD) methods often focus on transferring either feature, logic, or attention knowledge from teacher to student networks.
• Transferring only one type of knowledge leads to suboptimal performance due to the interconnected nature of reasoning processes and outcomes in neural networks.

Purpose of the Study:

• To investigate the impact of simultaneous knowledge transfer of reasoning processes and results in KD.
• To propose a novel KD method, Attention and Feature Transfer KD (AFT-KD), that effectively transfers both types of knowledge.

Main Methods:

• Developed attentional and feature blocks (AFBs) to encapsulate both inference process and outcome information from intermediate features.
• Employed block operations for efficient alignment between teacher and student networks to reduce computational overhead.
• Designed an adaptive loss function to dynamically balance the contribution of different loss components based on their optimization rates.

Main Results:

• The proposed AFT-KD method demonstrated superior performance compared to existing KD techniques.
• Experiments on multiple benchmark datasets confirmed the effectiveness of AFT-KD in improving student model accuracy.
• The simultaneous transfer of attention and feature knowledge proved crucial for enhanced distillation.

Conclusions:

• Simultaneously transferring knowledge related to both the reasoning process (attention) and reasoning results (features) is vital for effective knowledge distillation.
• AFT-KD offers a novel and effective approach to knowledge distillation, achieving state-of-the-art results.
• The method provides a new perspective for advancing research in knowledge distillation.