3D Scene Completion via Fusion of Diffusion Model Priors and Point Clouds

Three-dimensional (3D) scene reconstruction, essential for applications such as virtual reality, digital twins, and robotics, has been significantly advanced by Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, when visual information is missing or erroneous, it often results in incomplete scene reconstruction, featuring holes and artifacts in unseen regions. To address this, we aim to leverage point cloud priors covering unseen regions as reliable geometric cues. We propose a framework that combines diffusion priors with point cloud data for 3D scene completion under limited input conditions. In first stage, by leveraging the spatial information from point clouds combined with geometric optimization strategy, we effectively enhance the structural clarity of 2DGS scenes while significantly reducing rendering artifacts. Sparse depth maps projected from point clouds are used to train an image inpainting U-Net diffusion model, guiding the completion of unseen regions with geometrically consistent and stylistically accurate images. The proposed progressive completion strategy employs inpainted images to successively optimize the 2D Gaussian scenes. Experiments on multiple datasets show that our method reliably reconstructs unobserved regions with accurate scene structures and globally consistent styles under sparse input conditions. This approach extends the applicability of 3D reconstruction to real-world scenarios with limited visual data.