View Synthesis of Dynamic Scenes Based on Deep 3D Mask Volume

Area of Science:

• Computer Vision
• Deep Learning
• Computer Graphics

Background:

• Image view synthesis has advanced significantly with deep learning, but synthesizing dynamic scenes remains challenging.
• Lack of high-quality datasets and the complexity of the time dimension hinder progress in dynamic scene view synthesis.

Purpose of the Study:

• To introduce a novel multi-view video dataset for dynamic scene analysis.
• To develop a temporally-stable view extrapolation algorithm for dynamic scenes captured by static cameras.

Main Methods:

• A custom 10-camera rig captured 96 high-FPS dynamic outdoor scenes.
• The Deep 3D Mask Volume algorithm was developed for temporally-stable view extrapolation.
• The algorithm identifies and corrects error-prone disocclusion areas using 3D masks and static backgrounds.

Main Results:

• The proposed method demonstrates superior temporal stability compared to frame-by-frame synthesis and 2D mask methods.
• View synthesis videos exhibit minimal flickering artifacts.
• The algorithm allows for greater translational movements in synthesized views.

Conclusions:

• Deep 3D Mask Volume effectively addresses temporal inconsistencies in dynamic scene view synthesis.
• The new dataset and algorithm advance the field of immersive virtual experiences and dynamic scene reconstruction.