Enhancing Visual Grounding in Vision-Language Pre-Training With Position-Guided Text Prompts

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-Language Pre-Training (VLP) excels at aligning image-text pairs for cross-modal tasks.
• Current VLP models often lack robust visual grounding and localization, hindering performance in tasks like visual reasoning.

Purpose of the Study:

• To introduce a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding capabilities of VLP models.
• To improve object localization and understanding in cross-modal learning frameworks.

Main Methods:

• The PTP paradigm divides images into blocks and uses an object detector to identify objects within each block.
• It reframes visual grounding as a fill-in-the-blank task, predicting objects in blocks or block locations for objects.
• Second-order object relationships are integrated to further boost grounding performance.

Main Results:

• PTP integration led to significant improvements in VLP frameworks across multiple benchmarks.
• Notable gains include +5.6 in average recall@1 for Flickr30k Retrieval (ViLT baseline) and +5.5 in CIDEr for COCO Captioning (BLIP baseline).
• PTP achieves comparable results to object-detector-based methods with faster inference speeds by discarding the object detector during inference.

Conclusions:

• The Position-guided Text Prompt (PTP) paradigm effectively enhances visual grounding in VLP models.
• PTP offers a computationally efficient approach to improving cross-modal learning tasks, demonstrating strong performance and faster inference.