AttriPrompt: Class Attribute-Aware Prompt Tuning for Vision-Language Model.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Prompt tuning is an effective alternative to fine-tuning pre-trained vision-language models (VLMs).
• Existing class-shared and sample-specific prompt tuning methods struggle with imbalanced datasets, particularly underrepresented classes.
• Scarcity of data for tail classes hinders model performance in imbalanced scenarios.

Purpose of the Study:

• To propose an attribute-aware prompt tuning framework for balanced understanding in imbalanced tasks.
• To explicitly model critical class-level attributes to capture unique characteristics of underrepresented classes.
• To enhance the generalization of VLMs to underrepresented classes in imbalanced settings.

Main Methods:

• Developed an attribute pool to learn potential semantic attributes of classes using VLMs.
• Generated sample-specific prompts by selecting relevant attributes from the pool via a matching mechanism.
• Introduced a ProAdapter module to facilitate knowledge transfer and enhance generalization.

Main Results:

• The attribute-aware prompt tuning framework demonstrated superior performance, especially for tail classes.
• The proposed method effectively captures essential class semantics even for classes with limited data.
• Experiments on standard and imbalanced few-shot tasks validated the model's effectiveness.

Conclusions:

• Attribute-aware prompt tuning offers a promising approach for addressing class imbalance in VLM tasks.
• Explicitly modeling class attributes improves the handling of underrepresented classes.
• The framework enhances VLM performance and generalization in imbalanced few-shot learning scenarios.