DriveGen: Shared Video-Condition Encoding for Autonomous Multi-View Video Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Autonomous Systems

Background:

• Autonomous driving faces challenges with corner cases like severe weather and poor lighting.
• Collecting and annotating large datasets for these scenarios is expensive and time-consuming.
• Generative models offer a solution by expanding existing corner-case data.

Purpose of the Study:

• To develop a novel generative model for creating high-quality, controlled autonomous driving videos, specifically addressing challenges with multi-view data.
• To improve spatiotemporal consistency and annotation alignment in generated videos.
• To reduce the computational cost associated with generating diverse driving scenarios.

Main Methods:

• Proposed DriveGen, a model utilizing 4D position embeddings for multi-view video data.
• Implemented Dual-Scale Full Attention for global and local spatiotemporal consistency.
• Introduced a Shared Video-Condition Encoding (SVCE) Mechanism with a 3D VAE for efficient annotation encoding and pixel-level alignment.

Main Results:

• DriveGen achieved state-of-the-art performance in generating controlled autonomous driving videos.
• The model demonstrated improved global and local spatiotemporal consistency compared to existing methods.
• Achieved pixel-level alignment with significantly fewer learnable parameters (0.37M).

Conclusions:

• DriveGen effectively addresses the limitations of existing methods in generating multi-view autonomous driving videos.
• The proposed approach enhances generation quality, consistency, and computational efficiency.
• DriveGen shows significant promise for expanding corner-case datasets and advancing autonomous driving technology.