Dual-View Alignment Learning With Hierarchical-Prompt for Class-Imbalance Multi-Label Image Classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Real-world datasets frequently display class imbalance, leading to long-tailed distributions and few-shot scenarios.
• Class-Imbalanced Multi-Label Image Classification (CI-MLIC) tasks are particularly challenging due to data imbalance and the need for multi-object recognition.

Purpose of the Study:

• To propose a novel method, HP-DVAL, that leverages multi-modal knowledge from vision-language pretrained (VLP) models to mitigate class imbalance in multi-label image classification.
• To enhance the adaptability of VLP models for CI-MLIC tasks through a hierarchical prompt-tuning strategy.

Main Methods:

• HP-DVAL utilizes dual-view alignment learning to transfer feature representation capabilities from VLP models by extracting complementary features for image-text alignment.
• A hierarchical prompt-tuning strategy with global and local prompts is employed to learn task-specific and context-related prior knowledge.
• A semantic consistency loss is incorporated during prompt tuning to maintain general knowledge integrity within VLP models.

Main Results:

• The proposed HP-DVAL method demonstrated superior performance on two CI-MLIC benchmarks: MS-COCO and VOC2007.
• Significant improvements in mean Average Precision (mAP) were achieved: 10.0% and 5.2% on the long-tailed multi-label image classification task, and 6.8% and 2.9% on the multi-label few-shot image classification task, compared to state-of-the-art approaches.

Conclusions:

• HP-DVAL effectively addresses class imbalance in multi-label image classification by leveraging VLP models and a novel prompt-tuning strategy.
• The method shows significant potential for improving performance in challenging real-world scenarios characterized by data imbalance and few-shot learning.