Distribution-Aware Prompt Learning for Vision-Language Models With Dynamic Boundary Prototype.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Prompt learning adapts vision-language models (VLMs) using learnable prompts.
• Existing methods overlook atypical samples, limiting VLM generalization.
• Addressing this requires better calibration of visual feature space.

Purpose of the Study:

• To propose a novel framework, Distribution-Aware Prompt Learning (DAPL), for VLM adaptation.
• To enhance VLM generalization by addressing limitations in current prompt learning strategies.
• To improve the alignment between visual and textual representations by focusing on challenging samples.

Main Methods:

• Introduced dynamic boundary prototypes to highlight ambiguous samples.
• Developed Boundary-Centroid Pulling to optimize intra-class distribution.
• Designed a distance-weighted contrastive loss for inter-class separability.
• Applied Low-Rank Adaptation Fine-Tuning and a progressive training strategy.

Main Results:

• DAPL effectively calibrates the distribution of the visual feature space.
• Boundary-Centroid Pulling enhances intra-class structural consistency.
• The contrastive loss improves fine-grained discrimination between adjacent classes.
• DAPL consistently improves average performance when combined with existing prompt learning methods across 11 datasets.

Conclusions:

• DAPL offers a robust approach to enhance VLM generalization by considering sample distribution.
• The framework successfully addresses the limitations of focusing solely on representative samples.
• DAPL is compatible with existing prompt learning techniques, offering broad applicability and improved performance.