FISN: FInding Spatial Neighborhoods for Generalizable Novel View Synthesis.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Novel view synthesis aims to generate new images of a scene from a different viewpoint.
• Existing methods often struggle with efficiency and geometric consistency, especially for Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS).

Purpose of the Study:

• To develop a generalizable novel view synthesis algorithm enabling feedforward inference for NeRF and 3DGS.
• To address the challenges of high computational cost and degraded performance in existing approaches.

Main Methods:

• FISN integrates multi-reference 3D cost volumes into a unified 4D feature space.
• Introduces View-Spatial Convolutions for direct 4D feature aggregation and a Decomposable View-Spatial Convolution paradigm for efficiency.
• Incorporates Depth Refinement modules for improved global depth understanding.

Main Results:

• FISN achieves state-of-the-art performance on mainstream datasets for both NeRF and 3DGS.
• Demonstrates robustness in challenging scenarios, outperforming existing 3DGS-based methods.
• Maintains multiview geometric consistency while balancing efficiency and fine-grained spatial features.

Conclusions:

• FISN presents a significant advancement in generalizable novel view synthesis.
• The proposed method offers an efficient and robust solution for feedforward NeRF and 3DGS inference.
• FISN shows promise for applications requiring high-quality novel view generation from reference images.