Dual Balanced Class-Incremental Learning With im-Softmax and Angular Rectification.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Class incremental learning (CIL) methods, particularly exemplar-based approaches with knowledge distillation (KD), are prevalent due to their performance.
• Existing CIL methods struggle with data imbalance between old and new classes, leading to classifier bias towards new classes.
• Deep neural networks (DNNs) in CIL experience distribution drift, causing feature space narrowing and degraded representation of previously learned tasks.

Purpose of the Study:

• To address data imbalance in CIL by proposing an imbalance softmax (im-softmax) loss function.
• To mitigate feature space distribution drift and enhance old task representation using incremental-adaptive angular margin (IAAM) loss.
• To develop an integrated framework, dual balanced class incremental learning (DBL), combining im-softmax and IAAM for improved CIL performance.

Main Methods:

• Theoretical analysis of softmax loss inadequacy for imbalanced CIL data.
• Development of im-softmax loss by re-scaling output logits to underfit new classes.
• Implementation of IAAM loss to calibrate feature space by analyzing angle distributions between features and prototypes, recovering old feature representations.
• Integration of im-softmax and IAAM into an end-to-end DBL training framework.

Main Results:

• The proposed im-softmax loss effectively reduces bias in the linear classification layer caused by data imbalance.
• IAAM loss improves representation learning by reducing intra-class distance and enlarging inter-class margins, while recovering squeezed old feature distributions.
• The combined DBL framework achieves state-of-the-art (SOTA) performance on multiple benchmarks including CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet-100.

Conclusions:

• The dual balanced class incremental learning (DBL) framework effectively tackles key challenges in CIL: data imbalance and feature distribution drift.
• The novel im-softmax and IAAM loss functions provide significant improvements individually and synergistically within the DBL framework.
• The proposed method demonstrates superior performance and robustness across various standard CIL benchmarks.