Class-Incremental Continual Learning Into the eXtended DER-Verse.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks suffer from catastrophic forgetting, losing previously learned information when trained on new tasks.
• Class-Incremental Continual Learning (CICL) aims to enable models to learn sequentially without forgetting.
• Previous methods like Dark Experience Replay (DER) combine rehearsal and Knowledge Distillation but have limitations.

Purpose of the Study:

• To address the limitations of Dark Experience Replay (DER) in continual learning.
• To develop an enhanced continual learning method that improves knowledge retention and adaptability.
• To enable deep networks to learn new information without catastrophically forgetting past knowledge.

Main Methods:

• Introducing eXtended-Dark Experience Replay (X-DER), an advancement over DER.
• Implementing memory revision to incorporate novel information about past data.
• Developing mechanisms to facilitate the learning of previously unseen classes.
• Utilizing Knowledge Distillation and exploring the impact of flatter minima.

Main Results:

• X-DER demonstrates significant performance improvements over state-of-the-art methods.
• The proposed method achieves superior results on standard benchmarks like CIFAR-100 and miniImageNet.
• Performance gains are also observed on a newly introduced benchmark dataset.
• Ablation studies validate the effectiveness of Knowledge Distillation and flatter minima.

Conclusions:

• X-DER effectively overcomes catastrophic forgetting in deep networks.
• The method enhances continual learning by enabling memory revision and preparation for new classes.
• X-DER represents a significant advancement in Class-Incremental Continual Learning, offering improved performance and adaptability.