Decouple Before Align: Visual Disentanglement Enhances Prompt Tuning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Prompt tuning (PT) is a resource-efficient method for fine-tuning vision-language models.
• A key challenge in PT is information asymmetry between visual and textual modalities.
• This asymmetry can lead to biased attention, hindering model performance.

Purpose of the Study:

• To address the information asymmetry in prompt tuning for vision-language models.
• To propose a novel framework, DAPT, that enhances modal alignment and visual attention.
• To improve the task-specific transferability and generalization capabilities of these models.

Main Methods:

• Proposed DAPT (Decouple-Before-Align Prompt tuning) framework.
• Explicitly decoupled visual modality into foreground and background representations using segmentation cues.
• Aligned decoupled visual patterns with foreground texts and background classes.
• Introduced visual pull-push regularization for enhanced focus on region-of-interest objects.

Main Results:

• DAPT demonstrated superior performance across few-shot learning, base-to-novel generalization, and data-efficient learning benchmarks.
• The framework effectively mitigated biased attention caused by information asymmetry.
• Achieved enhanced modal alignment and improved visual concentration on relevant objects.

Conclusions:

• DAPT offers an effective, architecture-free solution for prompt tuning in vision-language models.
• The decouple-before-align strategy successfully addresses information asymmetry.
• The proposed methods significantly improve model performance and generalization capabilities.