Large Visual Language Models Continual Learning With Dynamic Mixture of Experts.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Continual learning is crucial for visual language models (VLMs) in dynamic environments.
• Existing methods struggle with scalability and performance degradation as task numbers increase.
• Catastrophic forgetting remains a significant challenge in continual learning for VLMs.

Purpose of the Study:

• To develop a novel continual learning framework for VLMs that addresses scalability and catastrophic forgetting.
• To enable VLMs to adapt to a growing number of open-set tasks while retaining historical knowledge.
• To reduce tunable parameters and improve the trade-off between model complexity and capacity.

Main Methods:

• The proposed framework builds upon a pre-trained CLIP model.
• It incorporates a dynamic mixture-of-experts (MoE) layer for flexible task adaptation.
• An elastic expert weight management strategy and adaptive-rank LoRA experts are employed to mitigate forgetting and optimize performance.

Main Results:

• The method demonstrates significant reduction in tunable parameters compared to existing approaches.
• It consistently outperforms state-of-the-art methods in learning new tasks.
• Performance on historical tasks is effectively maintained, overcoming catastrophic forgetting.

Conclusions:

• The proposed framework offers an effective solution for continual learning in VLMs.
• It enables VLMs to handle dynamic, open-set tasks efficiently and without knowledge loss.
• This approach advances the capabilities of VLMs in real-world, evolving application scenarios.