Sharing Task-Relevant Information in Visual Prompt Tuning by Cross-Layer Dynamic Connection.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Visual Prompt Tuning (VPT) shows promise for adapting pre-trained vision transformers.
• Existing VPT methods optimize prompts independently per layer, hindering cross-layer information sharing.
• Current prompt structures are susceptible to noise, impacting task-relevant information transfer.

Purpose of the Study:

• To propose SVPT, a novel VPT approach for improved information sharing and noise robustness.
• To enhance the adaptation of vision transformers to downstream tasks.
• To improve the flexibility and effectiveness of the prompt tuning process.

Main Methods:

• Introduced Cross-Layer Dynamic Connection (CDC) for prompt token information sharing between adjacent layers.
• Designed a Dynamic Aggregation (DA) module for selective inter-layer information sharing.
• Developed an Attentive Enhancement (AE) mechanism to refine salient image tokens with prompt tokens.

Main Results:

• SVPT demonstrated superior performance compared to state-of-the-art methods.
• Experiments on 24 benchmarks for image classification and semantic segmentation validated SVPT's effectiveness.
• The proposed CDC and DA modules enhanced attention flexibility and information sharing.

Conclusions:

• SVPT offers a significant advancement in visual prompt tuning by enabling effective cross-layer information sharing.
• The method enhances robustness against task-irrelevant noise, leading to better model adaptation.
• SVPT provides a more flexible and powerful approach for customizing vision transformers for diverse visual tasks.