Less confidence, less forgetting: Learning with a humbler teacher in exemplar-free Class-Incremental learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Class-Incremental Learning (CIL) is crucial for lifelong learning but suffers from catastrophic forgetting (CF).
• Knowledge Distillation (KD) is a common method to mitigate CF in CIL by using previous models as teachers.
• Exemplar-free CIL scenarios, lacking access to old data, exacerbate CF and pose unique challenges for KD.

Purpose of the Study:

• To investigate the over-confidence phenomenon of teacher models in exemplar-free CIL.
• To analyze the impact of Knowledge Distillation (KD) in scenarios without access to old training samples.
• To propose a novel approach to mitigate catastrophic forgetting (CF) by employing a more suitable teacher model.

Main Methods:

• Conducted empirical experiments and theoretical analysis to understand teacher over-confidence in exemplar-free CIL.
• Developed a new method, Learning with Humbler Teacher (LwHT), which selects an appropriate checkpoint model as a less over-confident teacher.
• Utilized nuclear norm regularization for temporal ensembling to enhance model stability.

Main Results:

• Identified and analyzed the over-confident behavior of teacher models in exemplar-free CIL.
• Demonstrated that LwHT significantly outperforms state-of-the-art methods by 10.41%, 6.56%, and 4.31% across different settings.
• Showcased superior model plasticity and stability compared to existing approaches.

Conclusions:

• The proposed Learning with Humbler Teacher (LwHT) effectively addresses catastrophic forgetting (CF) in exemplar-free Class-Incremental Learning (CIL).
• Selecting a 'humbler' teacher model and employing temporal ensembling enhances learning stability and performance.
• LwHT offers a promising direction for robust lifelong learning systems.