Real-Time High-Resolution View Synthesis of Complex Scenes With Explicit 3D Visibility Reasoning

Area of Science:

• Computer Graphics
• Computer Vision

Background:

• Rendering photo-realistic novel-view images of complex scenes is challenging.
• Existing view synthesis methods struggle with occlusion and real-time high-resolution rendering, especially for dynamic scenes with sparse views.

Purpose of the Study:

• To propose a generalizable view synthesis method for real-time, high-resolution novel-view image rendering.
• To address occlusion problems and improve rendering speed for complex static and dynamic scenes from sparse views.

Main Methods:

• Introduced an explicit 3D visibility reasoning approach to estimate point visibility to input views.
• Developed a fully differentiable visibility reasoning integrated into the volume rendering pipeline.
• Designed modules to bypass time-consuming MLP querying for enhanced high-resolution rendering.

Main Results:

• The method successfully renders high-resolution novel-view images of complex static and dynamic scenes in real-time from sparse views.
• Outperformed previous view synthesis methods in both rendering quality and speed.
• Demonstrated effective handling of occlusion problems inherent in sparse view scenarios.

Conclusions:

• The proposed method offers a significant advancement in view synthesis, particularly for challenging dynamic scenes.
• Achieves real-time, high-quality rendering by effectively integrating 3D visibility reasoning and optimized pipeline design.
• Provides a robust solution for generating novel views from limited input data.