Fine-Grained Visual Text Prompting.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Vision-Language Models (VLMs) excel at image-level understanding but lack precision in object localization and recognition.
• Existing visual prompting methods often incorporate irrelevant data, hindering performance.
• Effective collaboration between visual and text prompting for object-based tasks is underexplored.

Purpose of the Study:

• To introduce Fine-Grained Visual Text Prompting (FGVTP), a new zero-shot framework for object-based tasks.
• To enhance the performance of VLMs in precise object recognition and localization.
• To explore novel visual prompting designs and their synergy with text prompting.

Main Methods:

• FGVTP integrates Fine-Grained Visual Prompting (FGVP) and Consistency-Enhanced Text Prompting (CETP).
• FGVP utilizes precise semantic masks (e.g., from Segment Anything Model) and background blurring (Blur Reverse Mask) for target highlighting.
• CETP refines image-text alignment by generating captions based on FGVP-processed images.

Main Results:

• FGVTP achieves state-of-the-art (SOTA) zero-shot referring expression comprehension on RefCOCO/+/g benchmarks, outperforming prior methods by 5.8% on average.
• Experiments on the PACO dataset demonstrate FGVTP's superiority in part detection tasks.
• The proposed framework shows significant improvements in fine-grained visual understanding.

Conclusions:

• FGVTP offers a powerful new approach for zero-shot object-based vision-language tasks.
• Precise semantic masks and enhanced image-text alignment are key to improving VLM performance.
• The framework provides a robust solution for tasks requiring detailed object recognition and localization.