Logits -with-correlation-based distillation for class incremental learning with limited initial classes

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Class Incremental Learning (CIL) aims to learn new data classes without forgetting previous knowledge.
• Knowledge Distillation (KD) is a common strategy to mitigate forgetting in CIL, categorized into logit-based and feature-based methods.
• Feature-based KD performance significantly degrades with limited initial training classes, a common issue in domains like medical diagnosis.

Purpose of the Study:

• To propose an innovative Logits-with-Correlation-based Distillation (LCD) method for Class Incremental Learning.
• To address the performance sensitivity of KD-based CIL methods to the number of initial classes.
• To enhance both accuracy and robustness in CIL, particularly under data scarcity scenarios.

Main Methods:

• Developed Logits-with-Correlation-based Distillation (LCD) operating in logit space.
• Introduced an Inter-class Semantic Correlation-based less-forgetting Constraint.
• Incorporated an Intra-class Consistency Loss for feature space regularization.

Main Results:

• LCD achieves high accuracy by effectively alleviating forgetting and learning new classes.
• The proposed method demonstrates reduced sensitivity to the number of initial classes.
• Experiments show superior performance over state-of-the-art methods on natural and medical image datasets, especially with limited initial classes.

Conclusions:

• LCD offers a robust and accurate solution for Class Incremental Learning, overcoming limitations of existing KD approaches.
• The method's effectiveness is particularly pronounced in scenarios with constrained initial class data.
• LCD provides a promising direction for CIL applications requiring adaptability and resilience to data limitations.