TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding.

Area of Science:

• Computer Vision
• Robotics
• Virtual Reality

Background:

• Three-dimensional (3D) scene understanding relies on analyzing point cloud data for environmental perception.
• Existing methods align 2D image features with 3D point cloud features for open-vocabulary understanding.
• Current approaches face challenges with sparse/incomplete 3D data and lack direct text supervision during training.

Purpose of the Study:

• To address deficiencies in 3D feature extraction from sparse/incomplete point clouds.
• To improve the consistency between training and inference stages through direct text supervision.
• To enhance open-vocabulary 3D scene understanding capabilities.

Main Methods:

• Implemented a Masked Consistency training policy to handle sparse 3D features by masking parts of the data.
• Generated pseudo-text labels by creating scene descriptions from fused 2D image descriptions.
• Simultaneously aligned 2D-3D features and 3D-text features during the training process.

Main Results:

• The proposed method effectively addresses the challenges of sparse and incomplete 3D point cloud data.
• Direct text supervision during training improved the model's understanding and consistency.
• Experimental results demonstrate superior performance compared to state-of-the-art approaches.

Conclusions:

• The novel approach significantly advances 3D scene understanding by overcoming limitations of existing methods.
• The combination of masked consistency and pseudo-text supervision offers a robust solution for real-world scenarios.
• This work provides a more effective method for environmental perception in virtual reality, robotics, and beyond.