Learning Virtual View Selection for 3D Scene Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Joint 2D-3D learning is crucial for 3D vision tasks like semantic segmentation, leveraging complementary data.
• Current methods using only real 2D images suffer from redundancy, occlusion, and limited fields of view, hindering performance.
• Effective 3D scene understanding requires overcoming the limitations of standard 2D image inputs.

Purpose of the Study:

• To propose a general framework for joint 2D-3D scene understanding by selecting informative virtual 2D views.
• To improve 3D semantic segmentation by integrating generated virtual views with 3D geometry data.
• To enhance deep neural models for 3D vision tasks through a novel view selection strategy.

Main Methods:

• Generating virtual 2D views based on an information score map derived from 3D scene semantic segmentation results.
• Formalizing the information score map learning as a deep reinforcement learning process with rewards for accurate predictions.
• Employing an efficient greedy virtual view coverage strategy in 6D space (coordinates and normals) for optimal surface coverage.

Main Results:

• Validated the framework on ScanNet v2 and S3DIS datasets, demonstrating consistent gains over baseline models.
• Achieved new state-of-the-art accuracy for joint 2D and 3D scene semantic segmentation.
• The proposed method effectively improves performance for both joint 2D-3D and pure 3D input deep neural models.

Conclusions:

• The proposed virtual view selection framework significantly enhances 3D scene understanding.
• This approach effectively addresses the limitations of real-world 2D image data in 3D vision tasks.
• The method offers a general and effective solution for improving deep learning models in 3D semantic segmentation.