Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Vision Language Models (VLMs) excel at zero-shot tasks but struggle with specialized unseen data.
• Continual learning (CL) aims to adapt VLMs to new data without retraining, but faces catastrophic forgetting and generalization issues.
• Existing distillation methods for CL in VLMs are limited by single-teacher paradigms and inadequate use of multimodal data, increasing overhead.

Purpose of the Study:

• To address limitations in current distillation-based continual learning for VLMs.
• To propose a novel network, Multi-Stage Knowledge Integration (MulKI), inspired by Knowledge Integration Theory (KIT).
• To enhance VLM adaptation to evolving data distributions while preserving zero-shot capabilities.

Main Methods:

• Developed the Multi-Stage Knowledge Integration (MulKI) network, emulating human learning through four stages: Eliciting, Adding, Distinguishing, and Making Connections.
• Utilized prototypes for cross-modal alignment and constructed fine-grained intra- and inter-modality relationships.
• Adaptively distinguished and re-weighted knowledge from two teacher models, integrating preceding and new knowledge across tasks.

Main Results:

• MulKI demonstrated significant improvements in maintaining zero-shot capabilities during continual learning.
• The method effectively supports adaptation across diverse downstream tasks.
• MulKI mitigates catastrophic forgetting and generalization forgetting challenges inherent in CL.

Conclusions:

• The proposed MulKI network offers an effective approach to continual learning for Vision Language Models.
• MulKI successfully integrates knowledge by emulating human learning processes, overcoming limitations of existing methods.
• This work shows promise for adapting VLMs to dynamic, evolving data environments.