Single-View 3D Mesh Reconstruction for Seen and Unseen Categories

Area of Science:

• Computer Vision
• 3D Computer Graphics
• Machine Learning

Background:

• Single-view 3D object reconstruction from RGB images is a challenging computer vision problem.
• Existing deep learning methods struggle with generalizing to novel object categories not seen during training.

Purpose of the Study:

• To develop a method for single-view 3D mesh reconstruction that effectively generalizes to unseen object categories.
• To encourage models to reconstruct 3D objects more accurately and literally, irrespective of their training data.
• To break down category-specific limitations in 3D reconstruction.

Main Methods:

• Proposes GenMesh, an end-to-end two-stage network for single-view 3D mesh reconstruction.
• Factorizes the image-to-mesh mapping into image-to-point and point-to-mesh mappings, with the latter being less category-dependent.
• Employs a local feature sampling strategy in 2D and 3D spaces to capture shared geometric properties and enhance generalization.
• Introduces a multi-view silhouette loss for surface generation, supplementing traditional point-to-point supervision.

Main Results:

• GenMesh significantly outperforms existing methods on ShapeNet and Pix3D datasets.
• Demonstrates superior performance across various metrics and scenarios, particularly for reconstructing novel object categories.
• The proposed method shows enhanced generalization capabilities for unseen objects.

Conclusions:

• The proposed GenMesh network effectively addresses the generalization challenge in single-view 3D mesh reconstruction.
• The combination of stage factorization, local feature sampling, and multi-view silhouette loss leads to improved reconstruction accuracy for novel categories.
• This work advances the field by enabling more robust and versatile 3D shape recovery from single images.