VD-NeRF: Visibility-Aware Decoupled Neural Radiance Fields for View-Consistent Editing and High-Frequency Relighting

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Neural Radiance Fields (NeRFs) excel at novel view synthesis but struggle with scene manipulation due to integrated geometry and appearance encoding.
• Existing NeRF editing methods often produce view-inconsistent results and have limitations in modeling high-frequency lighting effects.
• Current lighting representations in NeRFs, such as Spherical Harmonics or low-resolution maps, are insufficient for complex relighting scenarios.

Purpose of the Study:

• To develop a visibility-aware approach (VD-NeRF) for decoupling view-independent and view-dependent appearance in 3D scenes.
• To enable consistent appearance editing and high-frequency environmental relighting in NeRFs.
• To improve the quality of scene decomposition and relighting by incorporating visibility information.

Main Methods:

• Reconstruct an explicit mesh from the input scene using a signed distance function.
• Employ a decoupled NeRF to attach view-independent appearance to the mesh using disentangled features.
• Utilize a hybrid lighting representation with an explicit learnable environment map and an implicit lighting network for comprehensive relighting.

Main Results:

• Achieved view-consistent results after editing scene appearance.
• Enabled high-frequency environmental relighting by supporting novel environment maps and an implicit lighting network.
• Demonstrated improved decomposition and relighting quality through visibility awareness.
• Outperformed previous methods quantitatively and qualitatively in editing and relighting tasks.

Conclusions:

• VD-NeRF successfully decouples scene properties for enhanced editability and relighting.
• The hybrid lighting representation and visibility awareness significantly improve rendering and editing outcomes.
• The proposed method offers a robust solution for complex 3D scene manipulation and realistic relighting applications.