Continual Learning With Knowledge Distillation: A Survey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Continual learning aims to enable AI models to learn sequentially without forgetting past knowledge.
• Catastrophic forgetting remains a primary challenge in developing adaptable AI systems.
• Knowledge distillation (KD) offers a regularization approach to mitigate forgetting by leveraging past model outputs.

Purpose of the Study:

• To comprehensively survey and analyze Knowledge Distillation (KD) methods in continual learning for image classification.
• To categorize KD applications in continual learning based on distinct paradigms and knowledge sources.
• To investigate the role and effectiveness of KD in consolidating memory within continual learning frameworks.

Main Methods:

• Systematic review and categorization of KD-based continual learning techniques.
• Analysis of KD's function from the perspective of loss functions in memory consolidation.
• Empirical evaluation of ten KD-integrated continual learning methods on CIFAR-100, TinyImageNet, and ImageNet-100 datasets.

Main Results:

• Knowledge distillation plays a critical role in mitigating catastrophic forgetting in continual learning.
• Classification bias negatively impacts KD effectiveness when combined with data replay.
• Utilizing a separated softmax loss function significantly enhances the efficacy of KD in continual learning.

Conclusions:

• KD is a vital component for effective continual learning, particularly in image classification tasks.
• The choice of loss function and integration with data replay strategies critically influence KD performance.
• Further research can leverage these findings to develop more robust and efficient continual learning systems.