Cost-Efficient Open Vocabulary 3D Scene Understanding Based on Semantic Probability.

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Scene Analysis

Background:

• Traditional 3D scene understanding relies on extensive 3D annotations, limiting recognition to seen classes.
• Existing open-vocabulary methods often incur high computational costs.
• Pre-trained models offer open-vocabulary inference but require adaptation for 3D tasks.

Purpose of the Study:

• To develop an efficient open-vocabulary 3D scene understanding framework.
• To enable recognition of unseen classes without 3D supervision or training.
• To reduce computational overhead compared to existing methods.

Main Methods:

• A semantic probability framework aligning CLIP text features with 2D pre-trained model pixel features.
• Projection of inferred semantic probabilities from pixels to 3D points.
• A point cloud fusion method to refine semantic probabilities and reduce errors.
• Leveraging semantic probabilities as pseudo-labels for 3D distillation.

Main Results:

• Achieved unsupervised 3D semantic segmentation on open vocabularies.
• Demonstrated competitive performance on ScanNet, Matterport3D, and nuScenes datasets.
• Showcased the potential of semantic probabilities for 3D distillation and performance improvement.

Conclusions:

• The proposed semantic probability framework offers an efficient and effective solution for open-vocabulary 3D scene understanding.
• The method successfully segments unseen classes without requiring 3D training data.
• This approach paves the way for more versatile and scalable 3D scene analysis.