Depth-Guided Optimization of Neural Radiance Fields for Indoor Multi-View Stereo

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Multi-view stereo (MVS) methods often struggle with pixel correspondence in complex indoor scenes.
• Neural Radiance Fields (NeRF) offer implicit scene representation but face challenges like shape-radiance ambiguity.

Purpose of the Study:

• To develop a novel multi-view depth estimation method, NerfingMVS, that leverages NeRF and learning-based priors.
• To address the limitations of existing methods by directly optimizing implicit volumes and mitigating NeRF's ambiguities.

Main Methods:

• NerfingMVS adapts a monocular depth network, fine-tuned on MVS reconstructions, to guide NeRF optimization.
• Depth priors are employed to monitor volume rendering, resolving shape-radiance ambiguity.
• A confidence map derived from rendering errors further refines depth estimation.

Main Results:

• NerfingMVS and its extension NerfingMVS++ achieve state-of-the-art results on ScanNet and NYU Depth V2 indoor datasets.
• The guided optimization improves depth quality without compromising NeRF's novel view synthesis capabilities.
• NerfingMVS++ introduces a coarse-to-fine prior training strategy and Gaussian sampling for enhanced performance.

Conclusions:

• NerfingMVS provides an effective approach for accurate multi-view depth estimation in indoor environments.
• The integration of learning-based depth priors significantly enhances NeRF-based reconstruction.
• The method demonstrates robust performance and maintains high-quality rendering for both seen and novel views.