Dual-modality adaptation in vision-language models for continual learning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Continual learning enables deep learning models to acquire new knowledge without forgetting previous information.
• Vision-language models (VLMs) are increasingly explored for continual learning.
• Current VLM continual learning methods often freeze one modality, limiting performance.

Purpose of the Study:

• To propose a novel continual learning framework for VLMs that fully leverages both image and text modalities.
• To overcome the limitations of single-modality adaptation in existing VLM continual learning approaches.

Main Methods:

• Implemented a dual-modality adaptation strategy using task-specific LoRA modules for the image encoder and class-specific learnable text prompts for the text encoder.
• Enhanced intra-class cohesion in image features with LoRA modules.
• Improved inter-class feature separation using learnable text prompts and a prompt-reuse training strategy.

Main Results:

• The proposed method significantly outperforms state-of-the-art approaches on multiple datasets.
• Demonstrated effective knowledge acquisition and catastrophic forgetting mitigation in continual learning scenarios.
• Showcased the benefits of dual-modality adaptation in VLMs.

Conclusions:

• The dual-modality adaptation framework effectively exploits the potential of large-scale pre-trained VLMs for continual learning.
• The proposed method offers a promising direction for advancing continual learning in multimodal AI.
• Future work will involve releasing the code for public access and further research.