Bi-Modality Individual-Aware Prompt Tuning for Visual-Language Model.

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning

Background:

• Prompt tuning adapts visual language models (VLMs) for tasks like domain generalization.
• Existing methods infer prompt tokens from training data, limiting adaptation to test data distributions.
• Context Optimization approaches improve generalization but struggle with test-set specific distributions.

Purpose of the Study:

• Introduce Bi-modality Individual-aware Prompt Tuning (BIP) to enhance VLM adaptability.
• Incorporate individual prior knowledge into learnable prompts for improved discriminability and generalization.
• Develop a novel approach for more effective adaptation of VLMs to diverse downstream tasks.

Main Methods:

• BIP utilizes Textual Knowledge Embedding (TKE) and Visual Knowledge Embedding (VKE) models.
• Projects class-aware textual knowledge into class-aware prompts (TCP) and instance-aware knowledge into instance-aware prompts (VIP).
• TCP dynamically adjusts classifiers for unseen domains; VIP enhances visual embedding discriminability.

Main Results:

• BIP demonstrated superior performance across 15 benchmarks in four distinct tasks.
• The method shows significant improvements in generalization and discriminative capabilities.
• Evaluations confirm BIP's effectiveness as a plug-and-play module.

Conclusions:

• BIP offers a novel and effective approach to prompt tuning for VLMs.
• The integration of individual prior knowledge significantly boosts model performance.
• BIP serves as a versatile and easily integrable module for existing VLM frameworks.