VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Computer Graphics

Background:

• Controllable image-to-video (I2V) generation requires precise control over camera, object motion, and lighting.
• Existing methods often fail to model the physical coupling between these factors, leading to visual inconsistencies.
• Lack of jointly annotated data hinders the development of models that can simultaneously control multiple factors.

Purpose of the Study:

• To present VidCRAFT3, a unified and flexible I2V framework for explicit modeling of cross-factor interactions.
• To enable independent and joint control over camera motion, object motion, and lighting direction.
• To address data scarcity through a novel dataset and progressive training strategy.

Main Methods:

• VidCRAFT3 integrates explicit 3D geometric priors (Image2Cloud) for camera motion control.
• Object motion is guided by multi-scale features from sparse object trajectories (ObjMotionNet).
• A Spatial Triple-Attention Transformer with cross-attention incorporates lighting direction for consistent relighting.

Main Results:

• VidCRAFT3 achieves state-of-the-art performance in control precision and visual coherence.
• The framework demonstrates robust learning without fully joint annotations.
• Experiments show improved consistency across diverse scenarios with simultaneous control.

Conclusions:

• VidCRAFT3 offers a unified approach to controllable I2V generation by modeling interdependencies.
• The framework enables high-fidelity video synthesis with precise control over multiple dynamic factors.
• The proposed dataset and training strategy facilitate advancements in the field.