Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge Distillation (KD) transfers information from large teacher to smaller student models.
• Existing KD methods often rely on task-specific knowledge, limiting generalizability.
• Self-supervised representation learning has shown recent success in improving feature learning.

Purpose of the Study:

• To propose a novel KD framework that leverages self-supervised learning for richer knowledge transfer.
• To enhance the learning of meaningful features in student networks beyond task-specific information.
• To improve the performance of KD through hierarchical feature map utilization.

Main Methods:

• Introduced an auxiliary self-supervision augmented task to guide feature learning.
• Derived soft self-supervision augmented distributions as enhanced 'dark knowledge'.
• Appended auxiliary branches to distill knowledge from hierarchical feature maps using self-supervised tasks.

Main Results:

• The proposed Hierarchical Self-Supervision Augmented KD (HSSAKD) achieved state-of-the-art performance in both offline and online settings on image classification.
• HSSAKD demonstrated improved feature learning capabilities.
• Transfer experiments on object detection validated the effectiveness of HSSAKD in guiding networks to learn better features.

Conclusions:

• HSSAKD effectively integrates self-supervised learning into KD for improved model compression.
• The method provides a robust approach to learning richer, more generalizable features.
• HSSAKD offers a promising direction for advancing knowledge distillation techniques.