VQ-NeRF: Neural Reflectance Decomposition and Editing With Vector Quantization.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Conventional neural reflectance fields utilize continuous representations for 3D scenes.
• This continuous approach can lead to noisy material decomposition and complex editing processes in reality.
• Objects in the real world are typically composed of discrete materials.

Purpose of the Study:

• To develop a novel method for discrete material editing in 3D scenes.
• To address limitations of continuous representations in neural reflectance fields.
• To enable intuitive and precise manipulation of materials within 3D environments.

Main Methods:

• Propose VQ-NeRF, a two-branch neural network incorporating Vector Quantization (VQ).
• A continuous branch predicts decomposed materials, while a discrete branch quantizes these into individual materials using VQ.
• Employ a dropout-based VQ codeword ranking strategy to determine the number of materials and reduce segmentation redundancy.

Main Results:

• VQ-NeRF successfully decomposes and quantizes continuous reflectance fields into discrete materials.
• The model generates a segmentation map, facilitating easy selection of specific materials for editing.
• Achieved superior performance in material decomposition and editing on both synthetic and real-world scenes.

Conclusions:

• VQ-NeRF is the first model to enable discrete material editing in 3D scenes.
• The proposed VQ mechanism significantly reduces noise and improves the usability of material editing.
• The two-branch architecture effectively bridges the gap between continuous representations and discrete real-world materials.