Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Scene Understanding

Background:

• Open-world instance-level scene understanding requires localizing and recognizing objects from categories not present in training data.
• Existing 2D methods benefit from large image-text datasets, but 3D scenarios lack sufficient 3D-text pairs, hindering progress.
• The scarcity of 3D-text data presents a significant challenge for training models capable of understanding novel 3D objects.

Purpose of the Study:

• To develop a method for open-world instance-level 3D scene understanding that addresses the lack of 3D-text data.
• To enable models to localize and semantically categorize novel 3D objects effectively.
• To improve the generalization capabilities of 3D instance grouping for accurate novel object localization.

Main Methods:

• Harnessing pre-trained vision-language (VL) foundation models to generate captions for multi-view 3D scene images.
• Implementing hierarchical point-caption association to learn semantic-aware embeddings from 3D geometry and multi-view images.
• Developing debiased instance localization using instance-level pseudo-supervision on unlabeled data for improved object grouping.

Main Results:

• Established explicit associations between 3D shapes and semantic-rich captions by generating captions for 3D scenes.
• Significantly enhanced fine-grained visual-semantic representation learning for object-level categorization.
• Achieved substantial performance improvements across 3D semantic, instance, and panoptic segmentation tasks on multiple datasets, outperforming baseline methods.

Conclusions:

• The proposed method effectively bridges the gap in 3D-text data scarcity by leveraging VL models for scene captioning.
• The hierarchical association and debiased localization techniques significantly boost the performance of open-world 3D scene understanding.
• The approach demonstrates strong generalization capabilities, paving the way for more comprehensive 3D perception systems.