Distilling Knowledge by Mimicking Features.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) trains efficient student networks using larger teacher networks.
• Traditional KD relies on soft logits from the teacher for supervision.
• There is a need for more effective KD methods that improve student model efficiency and accuracy.

Purpose of the Study:

• To propose a novel feature mimicking approach for knowledge distillation.
• To enhance the accuracy and efficiency of student networks.
• To investigate the role of feature magnitude and direction in distillation.

Main Methods:

• Student networks mimic features from the teacher's penultimate layer.
• A novel loss term based on locality-sensitive hashing (LSH) is introduced.
• Feature vectors are decomposed into magnitude and direction for targeted mimicking.

Main Results:

• The proposed feature mimicking method outperforms traditional KD in accuracy.
• LSH-based loss effectively guides students to mimic feature directions.
• The method achieves state-of-the-art distillation accuracy and extends to multi-label recognition and object detection.

Conclusions:

• Mimicking teacher features, particularly directions, is more effective than using soft logits.
• LSH provides a powerful mechanism for feature direction distillation.
• This approach offers a significant advancement in training efficient deep learning models.