Text to Image for Multi-Label Image Recognition With Joint Prompt-Adapter Learning.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Vision-language models (VLMs) like CLIP leverage image-text contrastive learning for parameter-efficient fine-tuning (PEFT).
• A significant challenge is the modality gap, limiting performance when using text as images (TaI).
• Multi-label image recognition (MLR) requires robust feature representation to handle multiple object classes within an image.

Purpose of the Study:

• To address the modality gap in VLMs for MLR using only text captions for PEFT.
• To introduce T2I-PAL, a novel method that utilizes text-to-image generation to bridge the modality gap.
• To enhance MLR performance and reduce the need for extensive manual annotation of training data.

Main Methods:

• Leveraging pre-trained text-to-image models to generate diverse, realistic images from text captions, reducing the text-image modality gap.
• Incorporating a class-wise heatmap and learnable prototypes to aggregate local similarities for robust visual feature representation.
• Combining prompt tuning and adapter learning for improved parameter-efficient fine-tuning (PEFT) and classification accuracy.

Main Results:

• T2I-PAL significantly reduces the modality gap between text and image representations.
• The method enhances the robustness and informativeness of local visual features for MLR.
• Experiments on MS-COCO, VOC2007, and NUS-WIDE benchmarks show an average performance boost of 3.47% over state-of-the-art methods.

Conclusions:

• T2I-PAL effectively tackles the modality gap in vision-language models for multi-label image recognition.
• The approach eliminates the need for fully semantically annotated training images, reducing manual annotation workload.
• T2I-PAL preserves the CLIP model's intrinsic mode, enabling seamless integration with existing CLIP frameworks and improving recognition performance.