PointDreamer: Zero-Shot 3D Textured Mesh Reconstruction From Colored Point Cloud.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Faithful reconstruction of textured 3D meshes is essential for diverse applications.
• Existing colored point cloud to mesh (colored-PC-to-mesh) methods often produce blurry textures or require extensive 3D training data.
• Leveraging 3D colored point clouds offers advantages for precise 360° object replication.

Purpose of the Study:

• To develop a novel framework, PointDreamer, for high-quality textured mesh reconstruction from 3D colored point clouds.
• To adapt 2D diffusion models for 3D point cloud data, overcoming limitations of existing methods.
• To address and mitigate unprojection artifacts in 3D reconstruction.

Main Methods:

• PointDreamer utilizes a novel project-inpaint-unproject pipeline to adapt 2D diffusion models.
• The framework projects point clouds into sparse 2D images for diffusion-based inpainting.
• High-quality textures are achieved by directly unprojecting inpainted 2D images to the 3D mesh, avoiding blurry 3D/UV space predictions.

Main Results:

• PointDreamer achieves state-of-the-art (SoTA) performance in textured mesh reconstruction, demonstrating a 30% improvement in LPIPS score (0.118 to 0.068).
• The proposed Non-Border-First (NBF) unprojection strategy effectively addresses unprojection artifacts at occlusion borders.
• The framework shows robustness to noisy, sparse, and incomplete input data, validated on synthetic and real-scanned datasets.

Conclusions:

• PointDreamer successfully harnesses 2D diffusion priors for superior textured mesh reconstruction from colored point clouds.
• The novel pipeline and NBF unprojection strategy overcome key limitations of prior colored-PC-to-mesh methods.
• PointDreamer offers a zero-shot, robust, and high-performance solution for 3D textured mesh generation.