Efficient semantic segmentation via logit-guided feature distillation

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Knowledge Distillation (KD) is crucial for model compression, transferring knowledge from large teacher models to smaller student models.
• Existing KD methods are categorized into logit distillation and feature distillation, with feature distillation previously outperforming.
• Deep Neural Networks (DNNs) may exhibit limited task-relevant features at early stages, hindering accuracy.

Purpose of the Study:

• To propose a Logit-guided Feature Distillation (LFD) framework combining logit and feature distillation for enhanced knowledge transfer.
• To improve semantic segmentation tasks by leveraging classification information from logits.
• To introduce a collaborative distillation method focusing on critical pixels and categories in early network stages.

Main Methods:

• Developed an LFD framework integrating logit and feature distillation.
• Utilized deep layer logits to generate fine-grained spatial masks for feature distillation, inducing spatial gradient disparities.
• Introduced class masks to dynamically adjust shallow auxiliary head weights and a novel shared auxiliary head distillation approach.

Main Results:

• The proposed LFD method achieves competitive performance on semantic segmentation tasks.
• The framework effectively transfers knowledge, enhancing student model capabilities.
• Experiments on Cityscapes, Pascal VOC, and CamVid datasets demonstrate the method's efficacy.

Conclusions:

• LFD offers an effective approach to model compression by synergizing logit and feature distillation.
• The method improves semantic segmentation accuracy while maintaining low memory footprint.
• The collaborative distillation strategy enhances early-stage feature learning and class-specific feature calibration.