DIST+: Knowledge Distillation From a Stronger Adaptive Teacher.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge distillation aims to transfer knowledge from a large teacher model to a smaller student model.
• Conventional methods struggle with significant prediction discrepancies between teacher and student models.
• Existing techniques often fail to capture intricate inter-class and intra-class relationships effectively.

Purpose of the Study:

• To introduce DIST, an innovative knowledge distillation method designed to overcome limitations of existing approaches.
• To enhance the learning process by effectively managing prediction discrepancies between teacher and student models.
• To improve the performance of distilled models across diverse computer vision tasks.

Main Methods:

• DIST employs a correlation-based loss to maintain prediction relationships and capture teacher's inter-class relations.
• The method incorporates intra-class similarity considerations, analyzing semantic similarities between instances and classes.
• Key enhancements include a teacher acclimation strategy and extending the DIST loss to the feature level.

Main Results:

• DIST successfully handles significant prediction discrepancies, outperforming conventional knowledge distillation techniques.
• The teacher acclimation strategy optimizes the distillation process by reducing teacher-student discrepancies.
• Extending DIST loss to the feature level significantly benefits dense prediction tasks like object detection and semantic segmentation.

Conclusions:

• DIST offers a simple, practical, and adaptable knowledge distillation solution suitable for various architectures and training strategies.
• The method consistently achieves state-of-the-art results in image classification, object detection, and semantic segmentation.
• DIST represents a significant advancement in knowledge distillation, enabling more effective model compression and performance enhancement.