SDPT: Synchronous Dual Prompt Tuning for Visual-Language Pre-trained Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Parameter-efficient fine-tuning is crucial for large pre-trained models.
• Existing prompt tuning methods struggle to maintain text-image alignment in dual-modal models.
• Visual-Language Pre-trained Models (VLPMs) require specialized adaptation techniques.

Purpose of the Study:

• To propose Synchronous Dual Prompt Tuning (SDPT) for effective VLPM adaptation.
• To preserve and leverage pre-trained text-image alignment during fine-tuning.
• To develop a parameter-efficient method for downstream tasks.

Main Methods:

• SDPT initializes unified prototype tokens in a shared modal aligning space.
• Learned prototype tokens represent aligned text and image semantics.
• Untrained inverse linear projections embed prototype information into modality-specific input spaces.

Main Results:

• SDPT achieves superior performance on downstream tasks for VLPMs.
• The method requires training only 0.04% of model parameters.
• SDPT outperforms existing single- and dual-modal prompt tuning approaches.

Conclusions:

• SDPT effectively adapts VLPMs while preserving crucial text-image alignment.
• The proposed method offers significant parameter efficiency.
• SDPT demonstrates strong performance across diverse downstream scenarios.