PERF: Panoramic Neural Radiance Field From a Single Panorama

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Neural Radiance Fields (NeRF) excel at novel view synthesis from multiple images.
• Existing single-image NeRF methods struggle with limited fields of view and occlusions, hindering real-world applications.
• Scalability to 360-degree panoramic scenes with significant occlusions remains a challenge.

Purpose of the Study:

• To develop a 360-degree novel view synthesis framework (PERF) capable of training a panoramic neural radiance field from a single panorama.
• To enable 3D roaming in complex scenes without the need for extensive multi-view image collection.
• To address limitations of existing methods in handling large occlusions and wide fields of view.

Main Methods:

• Proposed a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to reconstruct 3D scenes from 2D panoramas.
• Integrated an RGB Stable Diffusion model and a monocular depth estimator for completing RGB and depth maps.
• Developed a unified optimization framework incorporating these components for NeRF learning.

Main Results:

• PERF successfully reconstructs 3D scenes from single 360-degree panoramas, enabling 3D roaming.
• Demonstrated superior performance over state-of-the-art methods on Replica and the PERF-in-the-wild dataset.
• Achieved promising results in generating consistent geometry and realistic novel views.

Conclusions:

• PERF offers a scalable solution for 360-degree novel view synthesis from single panoramas.
• The framework facilitates 3D scene reconstruction and exploration in complex environments.
• PERF has potential for diverse real-world applications including panorama-to-3D, text-to-3D, and 3D scene stylization.