ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

Area of Science:

• Computer Vision
• 3D Reconstruction
• Generative Models

Background:

• Neural 3D reconstruction methods often require dense multi-view images, limiting their practical use.
• Existing techniques struggle with accurate camera pose control and limited view synthesis ranges.

Purpose of the Study:

• To develop a novel method, ViewCrafter, for generating high-fidelity novel views from single or sparse images.
• To leverage video diffusion models and 3D point-based representations for improved view synthesis.
• To enhance camera pose control accuracy and expand the range of generated novel views.

Main Methods:

• ViewCrafter utilizes a video diffusion model for powerful generation and point-based representations for coarse 3D information.
• A progressive view synthesis strategy expands the point cloud and coverage area for novel views.
• Integration with camera trajectory planning addresses occlusions and expands generation range.

Main Results:

• Achieved high-fidelity novel view synthesis with significantly improved camera pose control accuracy.
• Demonstrated strong generalization capabilities across diverse datasets.
• Enabled applications such as real-time rendering for immersive experiences and scene-level text-to-3D generation.

Conclusions:

• ViewCrafter overcomes the limitations of dense multi-view requirements in 3D reconstruction.
• The method offers a versatile solution for generating novel views, enhancing 3D content creation and immersive applications.
• The approach shows superior performance and generalization for high-fidelity novel view synthesis.