S $^{2}$ VG: 3D Stereoscopic and Spatial Video Generation via Denoising Frame Matrix.

Area of Science:

• Computer Vision
• 3D Graphics
• Artificial Intelligence

Background:

• Monocular video generation models produce high-quality videos but struggle with 3D and spatial video synthesis.
• Generating immersive 3D content for applications like virtual reality remains a significant challenge.

Purpose of the Study:

• To develop a pose-free and training-free method for generating immersive 3D videos from monocular videos.
• To adapt existing monocular video generation models for novel 3D and spatial video synthesis tasks.

Main Methods:

• A novel frame matrix inpainting framework is proposed, leveraging estimated depth information to warp monocular videos into pre-defined camera viewpoints.
• The original video generation model synthesizes missing content across viewpoints and timestamps, ensuring consistency without fine-tuning.
• A disocclusion boundary re-injection scheme is introduced to improve inpainting quality by mitigating latent space artifacts.

Main Results:

• The method successfully generates multi-view videos, adaptable into stereoscopic pairs or 4D Gaussians for spatial video synthesis.
• Experiments on videos from models like Sora, Lumiere, WALT, and Zeroscope show significant improvements over existing methods.
• The approach demonstrates robust performance across various generative models, enhancing 3D video creation.

Conclusions:

• The proposed method offers an effective and efficient solution for generating high-quality immersive 3D videos from monocular sources.
• This technique advances the creation of spatial and stereoscopic content, broadening applications in immersive technologies.
• The training-free and pose-free nature of the method makes it broadly applicable to various monocular video generation models.