Structured Knowledge Distillation for Dense Prediction.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Knowledge distillation is crucial for transferring knowledge from large to compact models.
• Existing methods for dense prediction tasks often adapt image classification distillation, leading to suboptimal performance.
• Dense prediction is inherently a structured problem, requiring specialized distillation strategies.

Purpose of the Study:

• To propose novel structured knowledge distillation methods for dense prediction tasks.
• To address the limitations of pixel-wise distillation in computer vision.
• To improve the performance of compact networks in tasks like semantic segmentation, depth estimation, and object detection.

Main Methods:

• Developed two structured distillation schemes: pair-wise distillation using static graphs and holistic distillation via adversarial training.
• Focused on distilling structural information, not just individual pixel predictions.
• Applied and evaluated methods on semantic segmentation, depth estimation, and object detection datasets.

Main Results:

• Demonstrated the effectiveness of the proposed structured distillation approaches.
• Achieved improved performance on multiple dense prediction tasks compared to existing methods.
• Validated the importance of considering the structured nature of dense prediction problems.

Conclusions:

• Structured knowledge distillation is a more effective approach for dense prediction tasks than traditional pixel-wise methods.
• The proposed pair-wise and holistic distillation schemes offer significant improvements for compact network design.
• This work provides a new direction for efficient model deployment in computer vision.