Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy.

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• High-accuracy semantic segmentation models are often computationally expensive, limiting their use in resource-constrained mobile applications.
• Knowledge distillation offers a method to transfer knowledge from large 'teacher' networks to smaller 'student' networks.
• Existing distillation methods often overlook channel relationships or can transfer incorrect knowledge from imperfect teacher models.

Purpose of the Study:

• To propose an efficient knowledge distillation method for training lightweight semantic segmentation networks.
• To address limitations in prior methods by incorporating both channel and spatial information, and by improving knowledge transfer accuracy.

Main Methods:

• Introduced a novel Channel and Spatial Correlation (CSC) loss function to capture comprehensive feature map relationships.
• Developed an Adaptive Cross Entropy (ACE) loss function to refine knowledge transfer by adaptively using ground truth and teacher predictions.
• Evaluated the proposed method on the Cityscapes and Camvid scene parsing datasets.

Main Results:

• The proposed CSC and ACE loss functions significantly improve the performance of lightweight semantic segmentation networks.
• The method outperforms previous knowledge distillation approaches in terms of accuracy and efficiency.
• Demonstrated the effectiveness of the approach for mobile vision applications.

Conclusions:

• The proposed knowledge distillation method effectively trains efficient semantic segmentation networks suitable for mobile applications.
• The novel loss functions enhance knowledge transfer by considering both feature correlations and adaptive label exploitation.
• This work advances the field of efficient deep learning for computer vision tasks.