DenseKD: Dense Knowledge Distillation by Exploiting Region and Sample Importance

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) compresses deep neural networks (DNNs) by transferring knowledge from teacher to student models.
• Cross-layer KD (CKD) enhances this process by distilling knowledge across network stages.
• Existing CKD methods suffer from improper channel alignment and uniform distillation, hindering student model performance.

Purpose of the Study:

• To propose DenseKD, a novel cross-layer knowledge distillation method.
• To address limitations in feature alignment and knowledge focus in current CKD techniques.
• To improve the efficiency and accuracy of compressed DNNs.

Main Methods:

• Developed a learnable dense architecture for flexible channelwise feature capture from the teacher model.
• Introduced region importance, using representation variations in teacher models to identify influential regions.
• Calculated sample importance based on teacher model loss to prioritize critical data samples during distillation.

Main Results:

• DenseKD demonstrated consistent improvements over state-of-the-art methods on various vision tasks.
• Achieved 72.30% accuracy with ResNet-20 on CIFAR-100 for classification, outperforming previous CKD approaches.
• Gained a 2.84% mean average precision (mAP) improvement for Faster R-CNN with ResNet-18 in object detection compared to vanilla KD.

Conclusions:

• DenseKD offers superior feature alignment and targeted knowledge transfer compared to existing CKD methods.
• The proposed approach effectively enhances student model performance in both classification and object detection tasks.
• DenseKD represents a significant advancement in efficient and accurate deep neural network compression.