Dual-modality adaptation in vision-language models for continual learning

Continual learning is a crucial capability for deep-learning models in real-world applications, enabling them to acquire new knowledge while avoiding catastrophic forgetting. With advances in large-scale pre-trained models, recent methods have begun exploring vision-language models (VLM) for continual learning scenarios. However, existing approaches primarily limit the training process to a single modality while keeping the other fixed for reference. This limits performance to the representational capacity of the fixed modality, thereby missing the integrated potential of both modalities. To address this issue, we propose a novel continual learning framework that fully leverages the pre-trained vision-language model by simultaneously incorporating task-specific LoRA modules for the image encoder and class-specific learnable text prompts for the text encoder. This dual-modality adaptation strategy fully exploits the potential of the large-scale pre-trained vision-language model. Specifically, the task-specific LoRA module facilitates the acquisition of new knowledge for new tasks to enhance the intra-class cohesion in image features. Additionally, using the pre-trained text encoder, we implement class-specific learnable text prompts to encourage feature separation across different classes. To further enhance inter-task class feature separation, we apply a training strategy that reuses previous text prompts and adopt a simple but effective inference strategy to better distinguish different classes. Extensive experiments across multiple datasets demonstrate that our method surpasses previous state-of-the-art approaches. The code will be released to the public.