Tuning Vision-Language Models With Multiple Prototypes Clustering.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Foundation models excel with large-scale pre-training but require domain-specific fine-tuning for expert performance.
• Vision-language models (VLMs) need effective few-shot tuning for specialized visual understanding tasks.
• Catastrophic forgetting and inefficient use of limited annotated data are key challenges in fine-tuning.

Purpose of the Study:

• To enhance the few-shot visual understanding capabilities of CLIP (Contrastive Language-Image Pre-training) models.
• To propose a novel adapter, lusterAdapter, that improves fine-tuning efficiency and performance.
• To mitigate catastrophic forgetting and maximize the utility of scarce annotated samples.

Main Methods:

• Developed lusterAdapter, a trainable adapter based on a multiple prototypes clustering algorithm.
• Incorporated anchors to preserve common knowledge and prevent catastrophic forgetting in foundation models.
• Utilized clustering and domain priors to enhance the efficiency of few annotated samples.

Main Results:

• Achieved state-of-the-art (SOTA) performance across 11 common classification benchmarks under few-shot settings.
• Demonstrated significant improvements over the original CLIP model, with up to a 19.6% accuracy increase in the 16-shot setting.
• Outperformed existing methods like TIP-Adapter and GraphAdapter by 2.7% and 2.2% respectively in average accuracy.

Conclusions:

• lusterAdapter effectively enhances few-shot visual understanding for CLIP models.
• The proposed method successfully addresses catastrophic forgetting and improves data efficiency.
• lusterAdapter represents a significant advancement in fine-tuning foundation models for specialized tasks.