Supervised contrastive learning with prototype distillation for data incremental learning

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Data Incremental Learning (DIL) aims to train models on non-stationary data streams without clear task divisions.
• Deep neural networks face catastrophic forgetting in DIL, hindering their ability to retain old knowledge while learning new information.
• Model stability and flexibility are crucial for effective DIL, requiring retention of past learning and adaptation to new data.

Purpose of the Study:

• To propose a novel method, Supervised Contrastive learning with Prototype Distillation (SCPD), to tackle the challenges of Data Incremental Learning.
• To enhance model stability and flexibility in DIL by mitigating catastrophic forgetting.
• To improve the performance of models in DIL scenarios, especially under imbalanced data distributions.

Main Methods:

• Utilizing supervised contrastive loss (SCL) to improve class separability and model flexibility.
• Introducing a prototype distillation loss (PDL) to enhance model stability by keeping feature representations close to class prototypes.
• Integrating SCL and PDL into the SCPD framework for comprehensive DIL.

Main Results:

• The proposed SCPD method demonstrated superior performance compared to state-of-the-art approaches.
• Experiments were conducted on several benchmarks, validating the effectiveness of SCPD.
• SCPD showed robust performance across various imbalanced data setups, highlighting its practical applicability.

Conclusions:

• SCPD effectively addresses catastrophic forgetting in Data Incremental Learning by balancing model stability and flexibility.
• The combination of supervised contrastive learning and prototype distillation offers a powerful solution for DIL.
• SCPD represents a significant advancement in incremental learning, outperforming existing methods and showing promise for real-world applications.