Learning to Reconstruct and Understand Indoor Scenes From Sparse Views.

Area of Science:

• Computer Vision
• Robotics
• Artificial Intelligence

Background:

• Current 3D reconstruction and semantic segmentation methods often require extensive data, such as videos from multiple cameras.
• Acquiring such data can be complex and limits the applicability of existing techniques.

Purpose of the Study:

• To develop a novel method for simultaneous 3D reconstruction and semantic segmentation of indoor scenes.
• To simplify data acquisition by using only a sparse set of uncalibrated color images.

Main Methods:

• An iterative deep architecture, IterNet, was designed to alternately estimate depth maps and semantic segmentation.
• A joint global and local registration method was proposed for accurate alignment of sparse views with limited overlap.
• A new indoor synthetic dataset with high-resolution RGB images, depth maps, and semantic labels was created.

Main Results:

• The proposed method achieves accurate depth estimation and semantic segmentation from sparse color images.
• The iterative approach effectively fuses depth maps for improved 3D scene reconstruction.
• Experimental results demonstrate superior performance compared to state-of-the-art methods on public and custom datasets.

Conclusions:

• The developed method offers a simplified approach to 3D reconstruction and semantic segmentation for indoor environments.
• The technique broadens the applicability of scene understanding by reducing data acquisition requirements.
• The IterNet architecture and registration method contribute to more accurate and robust 3D scene reconstruction with semantic information.