Frequency-Based Comprehensive Prompt Learning for Vision-Language Models.

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning

Background:

• Transformer-based Vision-Language Models (VLMs) show strong performance but lack discriminative local visual features.
• Existing methods struggle to extract representative local information from transformer visual encoders.
• Pre-trained VLMs require improved transferability for diverse downstream applications.

Purpose of the Study:

• To develop a method for learning comprehensive text prompts that describe visual concepts from coarse to fine.
• To enhance the transferability of pre-trained VLMs to various downstream tasks.
• To address the challenge of extracting local visual information from transformer-based VLMs.

Main Methods:

• Proposes Frequency-based Comprehensive Prompt Learning (FCPrompt) to extract local visual information.
• Utilizes Discrete Cosine Transform (DCT) to transform visual encoder features into the frequency domain.
• Employs a dual-branch framework with optimized text prompts for global and frequency-based local visual information alignment.

Main Results:

• FCPrompt effectively excavates representative local visual information from redundant features.
• The method obtains compact, informative, and disentangled local visual representations using DCT.
• Learned text prompts comprehensively describe visual concepts from coarse to fine.

Conclusions:

• FCPrompt significantly enhances the descriptive capabilities of VLMs.
• The approach achieves state-of-the-art performance across multiple benchmarks.
• The method provides a novel way to improve VLM adaptability and performance on downstream tasks.