Rethinking softmax in incremental learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Catastrophic forgetting is a major hurdle in incremental learning, where models forget previously learned information when trained on new data.
• The standard softmax cross-entropy distillation loss suffers from non-identifiability, hindering effective incremental learning.

Purpose of the Study:

• To propose novel strategies to mitigate catastrophic forgetting in incremental learning.
• To address the non-identifiability issue in softmax cross-entropy distillation loss.

Main Methods:

• Introduced an imbalance-invariant distillation loss to counteract imbalanced weights during distillation.
• Regularized prediction/distillation loss with shift-sensitive alternatives for problem identifiability.
• Developed five novel approaches integrating into existing frameworks like LWF, LWM, and LUCIR.

Main Results:

• Demonstrated consistent improvements in predictive accuracy across multiple incremental learning frameworks.
• Achieved substantial reductions in forgetting rates in extensive numerical experiments.
• On CIFAR-100, improved average accuracy by over 11% and reduced forgetting by over 16% for LWF, LWM, and LUCIR.

Conclusions:

• The proposed strategies effectively mitigate catastrophic forgetting in incremental learning.
• The novel approaches enhance the performance of distillation-based incremental learning methods.
• The research offers practical solutions for building more robust incremental learning systems.