Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos

Area of Science:

• Computer Vision
• Image Processing
• Artificial Intelligence

Background:

• Existing sky editing methods often focus on static images or require external sensor data.
• Dynamic sky replacement in videos presents challenges due to motion and lighting variations.

Purpose of the Study:

• To develop a purely vision-based, real-time framework for dynamic sky replacement and harmonization in videos.
• To eliminate the need for specialized capturing devices or inertial measurement units (IMUs).
• To enable both online and offline video processing for sky editing.

Main Methods:

• Decomposition of sky replacement into motion estimation, sky matting, and image blending.
• Development of a novel "flow propagation" method for robust motion estimation from video.
• Introduction of a coarse-to-fine sky matting network for accurate matte prediction.
• Implementation of an image blending technique for improved harmonization and content-aware image augmentation.

Main Results:

• The framework achieves real-time performance without manual interaction.
• Demonstrated high fidelity and generalization capabilities on diverse real-world videos.
• Showcased improvements in visual quality, lighting, and motion dynamics.
• Validated the benefit of content-aware image augmentation for autonomous driving perception.

Conclusions:

• The proposed vision-based framework offers an effective and versatile solution for dynamic sky replacement in videos.
• The method's independence from external sensors and real-time capability make it broadly applicable.
• The integration of advanced techniques like flow propagation and coarse-to-fine matting ensures high-quality results.