CKDF: Cascaded Knowledge Distillation Framework for Robust Incremental Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge distillation (KD)-based methods with exemplar rehearsal are common in class incremental learning (CIL).
• These methods face a feature uncalibration problem due to direct knowledge transfer, causing confusion in feature representations between old and new classes.
• This confusion leads to heterogeneous KD and classification losses, hindering effective knowledge transfer from previous tasks.

Purpose of the Study:

• To address the feature uncalibration problem in KD-based CIL methods.
• To propose a novel framework that calibrates existing knowledge and mitigates task-recency bias.
• To enhance the performance and generalizability of KD-based approaches in CIL.

Main Methods:

• Proposed a Feature Calibration Network (FCN) to calibrate existing knowledge and alleviate feature representation confusion.
• Introduced an image-feature hybrid sample rehearsal strategy to train FCN, overcoming memory limitations in CIL by storing lower-dimensional feature exemplars.
• Developed the Cascaded Knowledge Distillation Framework (CKDF) with three stages: FCN training, simultaneous new model training (KD and new class learning), and feature exemplar updating.

Main Results:

• The proposed CKDF framework effectively calibrates existing knowledge, reducing feature representation confusion.
• The hybrid rehearsal strategy allows for storing more samples, improving FCN training and mitigating task-recency bias.
• CKDF achieves state-of-the-art performance on several CIL benchmarks, demonstrating its effectiveness.

Conclusions:

• The CKDF framework offers a general solution applicable to various KD-based CIL methods.
• Feature calibration and hybrid rehearsal are crucial for improving performance in CIL.
• The proposed method significantly advances the capabilities of incremental learning systems.