Memory-Replay Knowledge Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Signal Processing

Background:

• Knowledge Distillation (KD) is crucial for Deep Neural Network (DNN) compression in intelligent sensor systems.
• Self-Knowledge Distillation (self-KD) methods often require task-specific DNN redesign or effective data augmentation.
• Existing self-KD approaches have limitations hindering widespread adoption.

Purpose of the Study:

• To introduce a novel self-KD method, Memory-replay Knowledge Distillation (MrKD), overcoming limitations of prior approaches.
• To leverage historical models as teachers within the self-KD framework.
• To enhance DNN training stability and performance without external knowledge dependencies.

Main Methods:

• Proposed a self-KD training strategy penalizing KL divergence between current and historical model outputs.
• Utilized a Fully Connected Network (FCN) to ensemble historical teacher outputs for guidance.
• Implemented Knowledge Adjustment (KA) to correct teacher logit outputs for ground truth accuracy.

Main Results:

• MrKD demonstrated improved single model training efficiency and performance.
• The method proved effective across diverse image (CIFAR-100, CIFAR-10, CINIC-10) and audio (DCASE) datasets.
• MrKD highlighted the value of utilizing historical models in DNN training.

Conclusions:

• MrKD offers an effective and practical self-KD approach by utilizing historical models.
• The method provides a stable learning regularization strategy through historical output distributions.
• MrKD presents a promising direction for DNN compression and performance enhancement.