Toward Fine-Grained 3-D Visual Grounding Through Referring Textual Phrases

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• 3D scene understanding is crucial for tasks like robotics and augmented reality.
• Current 3D visual grounding (3DVG) methods lack fine-grained contextual understanding.
• Existing 3DVG approaches overlook relationships between descriptive phrases and non-target objects.

Purpose of the Study:

• To introduce a more granular task, 3D phrase-aware grounding (3DPAG), enhancing 3D visual grounding.
• To develop methods for localizing target objects by identifying phrase-related objects and contextual reasoning.
• To create and utilize a comprehensive dataset for fine-grained phrase-aware 3D scene understanding.

Main Methods:

• Manual labeling of 227K phrase-level annotations from 88K sentences across Nr3D, Sr3D, and ScanRefer datasets.
• Development of a self-owned platform for efficient data annotation.
• Implementation of phrase-object alignment (POA) optimization and phrase-specific pretraining (PSP).

Main Results:

• Significant performance boosts in 3DVG by extending methods to the 3DPAG scenario.
• State-of-the-art methods achieved accuracy gains of 3.9% on Nr3D, 3.5% on Sr3D, and 4.6% on ScanRefer.
• Demonstrated the effectiveness of POA and PSP in improving fine-grained grounding capabilities.

Conclusions:

• 3DPAG offers a more detailed approach to 3D object localization compared to traditional 3DVG.
• The new datasets and methods advance the field of fine-grained 3D scene understanding.
• The released datasets and platform will facilitate further research in phrase-aware 3D visual grounding.