X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Rendering.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Generating photorealistic videos with precise control over scene elements remains a challenge.
• Existing methods often lack temporal consistency or intuitive editing capabilities.

Purpose of the Study:

• To introduce X2Video, a novel diffusion model for photorealistic video generation.
• To enable intuitive multi-modal control over video content using intrinsic channels and reference images/text prompts.
• To ensure temporal consistency and support detailed editing of video attributes.

Main Methods:

• Developed X2Video, a diffusion model extending image generation to video using intrinsic channels (albedo, normal, roughness, metallicity, irradiance).
• Employed Hybrid Self-Attention for temporal consistency and fidelity to reference images.
• Utilized Masked Cross-Attention for disentangling global and local text prompts.
• Introduced Recursive Sampling for generating long, temporally consistent videos via progressive frame sampling.

Main Results:

• X2Video successfully generates long, temporally consistent, and photorealistic videos guided by intrinsic conditions.
• The model effectively integrates multi-modal controls including reference images and global/local text prompts.
• Demonstrated parametric tuning for editing color, material, geometry, and lighting.

Conclusions:

• X2Video represents a significant advancement in controllable video generation.
• The model offers unprecedented flexibility in manipulating video content through intrinsic guidance and multi-modal inputs.
• The InteriorVideo dataset and X2Video model will be publicly released.