JIMR: Joint Semantic and Geometry Learning for Point Scene Instance Mesh Reconstruction.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Geometric Deep Learning

Background:

• Point scene instance mesh reconstruction is complex, requiring simultaneous instance segmentation and mesh generation from partial data.
• Existing methods often use separate detection or segmentation backbones, limiting mesh quality.

Purpose of the Study:

• To develop a joint framework (JIMR) for improved 3D point scene instance mesh reconstruction.
• To enhance both local details and global smoothness in reconstructed meshes.

Main Methods:

• Proposed a two-stage cascaded framework for semantic and geometric understanding.
• Stage 1: Simultaneous instance segmentation and object detection to improve point accuracy and alignment.
• Stage 2: A 'complete-then-reconstruct' module disentangling completion and reconstruction, leveraging pre-trained networks and a confidence score for proposal filtering.

Main Results:

• The joint optimization in Stage 1 provides more precise instance points and better alignment.
• The disentangled completion and reconstruction in Stage 2 allow for modular improvements.
• The comprehensive confidence score effectively filters low-quality reconstructions.

Conclusions:

• JIMR significantly outperforms state-of-the-art methods in 3D point scene instance mesh reconstruction.
• The proposed joint framework effectively addresses the challenges of simultaneous segmentation and reconstruction.
• JIMR demonstrates superior qualitative and quantitative results in reconstructing detailed and smooth 3D meshes.