Intra-class progressive and adaptive self-distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks face challenges with increasing parameters, leading to high computational load and training time.
• Knowledge Distillation (KD) is a key technique for model compression, training efficient student models.
• Existing self-distillation methods (offline-KD, online-KD) often neglect crucial feature and category information.

Purpose of the Study:

• To introduce a novel self-distillation method, Intra-class Progressive and Adaptive Self-Distillation (IPASD), for improved model compression.
• To address the limitations of existing self-distillation techniques by incorporating feature-level and category information.
• To enhance the efficiency and reduce the computational cost of deep neural networks.

Main Methods:

• IPASD transfers knowledge progressively between adjacent epochs, focusing on intra-class feature extraction and compactness.
• The method integrates feature-level and logits-level knowledge, leveraging a strong teacher's knowledge.
• Adaptive optimization is achieved using ground-truth labels as supervision signals.

Main Results:

• IPASD demonstrated superior performance compared to state-of-the-art self-distillation methods.
• The method showed significant improvements in knowledge transfer and model compression.
• Evaluations were conducted on diverse datasets including CIFAR-10, CIFAR-100, Tiny ImageNet, Plant Village, and ImageNet.

Conclusions:

• IPASD offers an effective approach to self-distillation, enhancing model compression and knowledge transfer.
• The method's ability to extract class-typical features and promote intra-class compactness contributes to its success.
• IPASD represents a significant advancement in efficient deep learning model training.