Volume Feature Aware View-Epipolar Transformers for Generalizable NeRF.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Generalizable Neural Radiance Fields (NeRF) synthesize novel views without per-scene training.
• View-epipolar transformers are popular for high-quality view synthesis in NeRF.
• Existing methods assume texture consistency for surface identification, which can be ambiguous.

Purpose of the Study:

• To address ambiguity in surface identification within view-epipolar transformers for NeRF.
• To improve the accuracy of surface detection and subsequent novel view synthesis.
• To enhance the performance of generalizable NeRF models.

Main Methods:

• Introduced 3D volume features to supplement texture features in view-epipolar transformers.
• Integrated both texture and geometric cues for more robust consistency measurement.
• Proposed a decoupled decoder for separate density and color prediction using volume and texture features, respectively.

Main Results:

• Mitigated ambiguity in surface detection by combining texture and geometric information.
• Achieved more accurate surface reconstruction, leading to improved novel view synthesis.
• Demonstrated superior performance compared to existing transformer-based methods on diverse datasets.

Conclusions:

• The integration of 3D volume features significantly enhances the capabilities of view-epipolar transformers in generalizable NeRF.
• The proposed method offers a more robust approach to novel view synthesis by overcoming texture-based ambiguities.
• The decoupled decoder design further optimizes the prediction of scene properties.