Imitating the oracle: Towards calibrated model for class incremental learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Class-incremental learning (CIL) enables models to learn new classes over time without forgetting previous ones.
• Joint-training (JT) serves as an upper bound for CIL performance by training on all classes simultaneously.
• Existing CIL methods often struggle with catastrophic forgetting and performance degradation on older classes.

Purpose of the Study:

• To bridge the performance gap between CIL and JT by proposing a novel method to imitate JT.
• To analyze the differences between CIL and JT in feature and weight spaces to inform method development.
• To introduce a plug-and-play solution that enhances existing CIL algorithms.

Main Methods:

• Feature calibration: Compensates for deviations to preserve class decision boundaries in feature space.
• Weight calibration: Employs forgetting-aware weight perturbation to enhance transferability and reduce parameter forgetting.
• Imitate the Oracle (ItO): A novel approach combining feature and weight calibration to mimic JT properties.

Main Results:

• The proposed ItO method significantly improves performance across various benchmark CIL datasets.
• ItO consistently enhances the effectiveness of existing state-of-the-art CIL methods.
• Experiments validate the efficacy of both feature and weight calibration strategies.

Conclusions:

• The ItO method effectively addresses limitations in CIL by successfully imitating joint-training.
• Feature and weight calibration are crucial for mitigating forgetting and improving knowledge transfer in incremental learning.
• ItO offers a practical and broadly applicable enhancement for CIL systems.