Model Behavior Preserving for Class-Incremental Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep models suffer from catastrophic forgetting, degrading performance on old data after fine-tuning on new data.
• Knowledge distillation (KD) is a common method to mitigate this, but often ignores the structural relationships within data instances.
• Convolutional Neural Network (CNN) models require methods that consider global instance properties for effective incremental learning.

Purpose of the Study:

• To develop a novel approach to address catastrophic forgetting in deep models.
• To preserve the intrinsic structure of neural network responses during incremental learning.
• To improve recognition performance on old data after fine-tuning on new data.

Main Methods:

• Designed an instance neighborhood-preserving (INP) loss to maintain pairwise instance similarities in the feature space.
• Devised a label priority-preserving (LPP) loss to preserve label ranking within output probability vectors.
• Introduced an efficient derivable ranking algorithm for calculating the proposed loss functions.

Main Results:

• The proposed approach effectively preserves the order of pairwise instance similarities.
• Label ranking lists within instance-wise probability vectors are successfully maintained.
• Achieved state-of-the-art performance on CIFAR100 and ImageNet datasets.

Conclusions:

• The novel INP and LPP losses offer a significant advancement in combating catastrophic forgetting.
• Considering global instance properties and label rankings is crucial for effective incremental learning.
• The developed methods provide a robust solution for enhancing the stability of deep models in incremental learning scenarios.