Aligning to the teacher: multilevel feature-aligned knowledge distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation aims to transfer knowledge from large teacher models to smaller student models.
• Existing methods often suffer from poor distillation effects due to feature differences and limited student generalization.
• Teacher models possess richer features than student models, hindering effective knowledge transfer.

Purpose of the Study:

• To propose a novel Multilevel Feature Alignment Knowledge Distillation (MFAKD) method.
• To reduce feature differences between teacher and student models for improved knowledge transfer.
• To enhance the generalization ability and performance of student models.

Main Methods:

• Implemented a spatial dimension alignment module using upsampling to match student and teacher feature maps.
• Introduced a multibranch channel alignment (MBCA) module to align features across channels.
• Utilized discriminative classifiers from the pretrained teacher model for student inference.

Main Results:

• Achieved state-of-the-art results on the CIFAR-100 dataset.
• The student model WRN-40-2 exceeded the teacher model ResNet-8×4 accuracy by nearly 2%.
• Demonstrated excellent performance on the Tiny ImageNet dataset, validating the method's effectiveness.

Conclusions:

• MFAKD effectively reduces feature discrepancies between teacher and student models.
• The proposed method enables student models to learn comprehensive features and surpass teacher performance.
• MFAKD offers a promising approach for efficient and effective knowledge distillation in deep learning.