PiGLET: Pixel-Level Grounding of Language Expressions With Transformers

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• The natural language visual grounding problem aims to connect textual descriptions to specific regions in images.
• Existing methods often lack the spatial granularity and generality required for complex scene understanding.

Purpose of the Study:

• To introduce Panoptic Narrative Grounding, a spatially fine-grained and general formulation of visual grounding.
• To propose PiGLET, a novel multi-modal Transformer architecture for this task.
• To establish an experimental framework with new ground truth and metrics for evaluation.

Main Methods:

• Developed an algorithm to automatically transfer annotations to panoptic segmentations.
• Proposed PiGLET, a multi-modal Transformer leveraging panoptic categories and fine-grained segmentations.
• Evaluated PiGLET on the MS COCO dataset for Panoptic Narrative Grounding.

Main Results:

• PiGLET achieved 63.2 absolute Average Recall points on the Panoptic Narrative Grounding task.
• PiGLET improved panoptic segmentation by 0.4 Panoptic Quality points on the MS COCO benchmark.
• Demonstrated generalizability to other visual grounding tasks like Referring Expression Segmentation.

Conclusions:

• Panoptic Narrative Grounding offers a more comprehensive approach to visual grounding.
• PiGLET is an effective architecture for this task and shows competitive performance on existing benchmarks.
• The proposed framework facilitates future research in fine-grained, general visual grounding.