NeRF-MIR: Toward High-Quality Restoration of Masked Images With Neural Radiance Fields

Area of Science:

• Computer Vision
• 3D Scene Reconstruction
• Neural Rendering

Background:

• Neural Radiance Fields (NeRFs) excel at novel view synthesis but struggle with corrupted or masked image data.
• Effective restoration of 3-D scenes from incomplete visual information is crucial for real-world applications.

Purpose of the Study:

• To develop a novel neural rendering approach, NeRF-MIR, for effective masked image restoration in 3-D scenes.
• To enhance NeRF's capability in reconstructing complete 3-D representations from incomplete visual data.

Main Methods:

• Proposed a patch-based entropy for ray emitting (PERE) strategy to optimize ray distribution for better texture learning.
• Introduced a progressively iterative restoration (PIRE) mechanism for self-training and refining masked regions.
• Designed a dynamically weighted loss function to adaptively adjust restoration priorities.

Main Results:

• NeRF-MIR demonstrated superior performance in masked image restoration compared to existing methods.
• Experiments on real and newly constructed masked datasets validated the effectiveness of the proposed approach.
• The PERE strategy and PIRE mechanism significantly improved the fusion of multi-view information and the quality of restored regions.

Conclusions:

• NeRF-MIR offers a powerful solution for 3-D scene restoration from masked images, showcasing NeRF's potential beyond novel view synthesis.
• The developed methods address key limitations in NeRF's ability to handle corrupted input data.
• This work contributes new datasets and methodologies for the emerging field of NeRF-based image restoration.