MSP: Multimodal Self-Attention Prompt Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal prompt learning adapts vision-language models (VLMs) like CLIP to new tasks.
• Conventional methods adapt VLMs at the input level, causing geometric distortions and limiting generalization.
• This indirect adaptation leads to overfitting and reduced performance on novel classes, known as the indirect learning dilemma (ILD).

Purpose of the Study:

• To introduce a novel framework, Multimodal Self-Attention Prompt (MSP), for efficient and generalizable VLM adaptation.
• To overcome the indirect learning dilemma (ILD) by shifting adaptation from input to semantic layers.
• To enhance the performance and robustness of VLMs on downstream tasks, particularly for novel classes.

Main Methods:

• Developed the Multimodal Self-Attention Prompt (MSP) framework.
• Injected learnable prompts directly into the key and value sequences of Transformer attention blocks.
• Incorporated distance-aware optimization and partial prompt learning (stochastic dimension masking) for improved adaptation and robustness.

Main Results:

• Achieved state-of-the-art harmonic mean accuracy of 80.67% across 11 benchmarks.
• Demonstrated superior performance on novel classes with 77.32% accuracy, a 2.18% absolute improvement over prior methods.
• Required only 0.11M learnable parameters and surpassed CLIP's zero-shot performance on 10 out of 11 datasets.

Conclusions:

• The MSP framework effectively addresses the indirect learning dilemma in multimodal prompt learning.
• Directly modulating attention mechanisms enables precise downstream adaptation while preserving pretrained embeddings.
• MSP establishes a new paradigm for efficient, robust, and generalizable prompt-based adaptation of VLMs.