Point Cloud Scene Completion With Joint Color and Semantic Estimation From Single RGB-D Image

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Reconstruction

Background:

• Severe occlusion in single RGB-D images poses challenges for 3D scene reconstruction.
• Existing methods struggle with completing large missing areas and maintaining semantic consistency.

Purpose of the Study:

• To develop an end-to-end deep reinforcement learning method for high-quality colored semantic point cloud scene completion.
• To address severe occlusion and reconstruct complete 3D scenes from limited input data.

Main Methods:

• A novel progressive view inpainting approach guided by 3D scene volume reconstruction.
• Integration of 2D RGB-D and semantic segmentation map inpainting.
• Utilizing an A3C network for optimal multi-view selection to progressively fill occluded regions.

Main Results:

• Achieved high-quality scene reconstruction from single, heavily occluded RGB-D images.
• Demonstrated robust and consistent results through joint learning of all modules.
• Outperformed state-of-the-art methods in qualitative and quantitative evaluations on the 3D-FUTURE dataset.

Conclusions:

• The proposed deep reinforcement learning method effectively completes occluded 3D scenes.
• Volume guidance and progressive view inpainting are crucial for handling severe occlusion.
• The approach offers a significant advancement in single-view 3D scene reconstruction.