Dual Adapter Tuning of Vision-Language Models Using Large Language Models.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-language models (VLMs) excel in zero-shot tasks but can be improved with limited data.
• Efficient transfer learning (ETL) using feature adapters is key for VLM adaptation.
• Existing ETL methods often focus on single adapter types or generic prompts.

Purpose of the Study:

• To propose a novel ETL approach for VLMs that combines multiple adapter styles.
• To improve prompt generation for VLMs using attribute-specific, LLM-generated prompts.
• To enhance VLM discriminative capabilities with context-aware information.

Main Methods:

• Developed a novel ETL approach leveraging both prior-independent and prior-dependent feature adapters simultaneously.
• Utilized a pre-trained large language model (LLM) to generate attribute-specific prompts for visual categories.
• Incorporated context-aware discriminative information from LLM to guide VLM classification.

Main Results:

• The proposed ETL model achieved state-of-the-art performance across 11 diverse datasets.
• Simultaneous use of adapter styles and LLM-generated prompts significantly boosted VLM transferability.
• Attribute-specific and context-aware prompts improved the model's ability to distinguish between classes.

Conclusions:

• The novel ETL approach offers a significant advancement in adapting VLMs with limited data.
• LLM-driven prompt engineering is effective for enhancing VLM performance in transfer learning.
• The method sets a new benchmark for efficient transfer learning in vision-language tasks.