Mixture of prompts learning for vision-language models.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-language models (VLMs) like CLIP are powerful for downstream tasks.
• Prompt learning is an effective adaptation method for VLMs, requiring few parameters.
• Existing prompt learning methods struggle with dataset diversity and overfitting.

Purpose of the Study:

• To address limitations in single soft prompt adaptation for VLMs.
• To improve the adaptability and generalization capabilities of VLMs.
• To mitigate overfitting issues in prompt fine-tuning.

Main Methods:

• Proposed a mixture-of-prompts learning method with a routing module for dynamic prompt selection.
• Introduced a gating mechanism for prompt selection based on hard prompt similarity.
• Implemented semantically grouped text-level supervision with contrastive loss for knowledge preservation.

Main Results:

• Demonstrated significant improvements on 11 datasets across few-shot learning, domain generalization, and base-to-new generalization.
• Outperformed existing baseline methods in adapting VLMs to new tasks.
• Validated the effectiveness of multi-prompt specialization and knowledge-preserving routing.

Conclusions:

• The proposed method effectively bridges the adaptability-generalization tradeoff in VLM deployment.
• Mixture-of-prompts learning with routing enhances VLM performance and robustness.
• This approach offers a more effective way to adapt powerful pre-trained VLMs.