General 3D Vision-Language Model With Fast Rendering and Pre-Training Vision-Language Alignment

Area of Science:

• Computer Vision
• Machine Learning
• 3D Scene Understanding

Background:

• Deep neural networks for 3D scene understanding typically require extensive labeled data and struggle with novel object categories.
• Current methods face limitations in recognizing unseen classes and often perform poorly with scarce labels.

Purpose of the Study:

• To develop a generalized framework for 3D point cloud segmentation and detection that performs effectively with limited labeled data.
• To enable open-vocabulary 3D scene understanding, allowing recognition of novel categories beyond the training set.

Main Methods:

• A hierarchical feature-aligned pre-training and knowledge distillation strategy to leverage large-scale vision-language models.
• An energy-based loss function incorporating boundary awareness for improved region-level predictions.
• An unsupervised region-level semantic contrastive learning scheme for point cloud instance discrimination.

Main Results:

• WS3D++ achieved state-of-the-art results on the ScanNet benchmark for semantic and instance segmentation with limited data.
• Demonstrated superior data-efficient learning performance on S3DIS and SemanticKITTI datasets for both indoor and outdoor scenes.
• Validated effectiveness in open-world few-shot learning scenarios through extensive experiments.

Conclusions:

• The proposed WS3D++ framework effectively addresses the challenge of limited labeled data in 3D scene understanding.
• The approach facilitates open-vocabulary recognition and achieves state-of-the-art performance in data-efficient and few-shot learning settings.
• The publicly available code and models will benefit future research in 3D point cloud analysis.