Learning Without Forgetting for Vision-Language Models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Class-Incremental Learning (CIL) aims for systems to learn new tasks without forgetting previous ones.
• Vision-Language Models (VLMs) show potential for generalizable representations but suffer catastrophic forgetting in CIL.
• Applying VLMs to CIL faces challenges in preventing knowledge loss and leveraging multi-modal information.

Purpose of the Study:

• To develop a novel method enabling VLMs to perform CIL without catastrophic forgetting.
• To effectively utilize multi-modal information within VLMs for improved continual learning.
• To address the dual challenges of knowledge retention and cross-modal fusion in CIL.

Main Methods:

• Proposed PROjectiOn Fusion (Proof) framework for CIL with VLMs.
• Implemented task-specific projections on frozen image/text encoders, expanding for new tasks and fixing old ones.
• Introduced a fusion module to jointly adjust visual and textual features for enhanced semantic understanding.

Main Results:

• Proof significantly alleviates forgetting of former knowledge during incremental training.
• The fusion module effectively captures task-specific semantic information by integrating cross-modal features.
• Achieved state-of-the-art performance across nine benchmark datasets and various CIL scenarios.

Conclusions:

• Proof offers an effective solution for enabling VLMs in Class-Incremental Learning.
• The proposed projection and fusion strategies enhance model adaptability and multi-modal utilization.
• This work advances the capability of AI systems to learn continually and robustly.