OutDreamer: Video Outpainting With a Diffusion Transformer

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video outpainting extends video content beyond original boundaries, demanding temporal and spatial consistency.
• Existing latent diffusion models with U-Net backbones face challenges in quality and adaptability.
• Diffusion transformers (DiTs) show superior performance, offering a promising alternative.

Purpose of the Study:

• To introduce OutDreamer, a Diffusion Transformer (DiT)-based framework for high-quality video outpainting.
• To enhance adaptability and consistency in generated video content.
• To outperform existing video outpainting methods on benchmarks.

Main Methods:

• Developed OutDreamer, a DiT-based framework with video control and conditional outpainting branches.
• Introduced a mask-driven self-attention layer for dynamic mask integration.
• Implemented a latent alignment loss for frame-level consistency.
• Utilized a cross-video-clip refiner for temporal consistency in long videos.

Main Results:

• OutDreamer demonstrates superior performance in video outpainting tasks.
• The framework achieves high quality and adaptability in generated video content.
• Extensive evaluations confirm OutDreamer's effectiveness on benchmarks.

Conclusions:

• OutDreamer significantly advances video outpainting capabilities using Diffusion Transformers.
• The proposed methods ensure both spatial and temporal consistency in extended video content.
• OutDreamer represents a state-of-the-art approach for video content generation.