Learning Dynamic Textures for Neural Rendering of Human Actors.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Neural network-based human video synthesis offers efficiency over traditional graphics rendering.
• Current methods often struggle with artifacts like over-smoothing, missing body parts, and temporal instability in fine details.

Purpose of the Study:

• To develop a novel human video synthesis method that overcomes limitations of existing 2D image-to-image translation approaches.
• To explicitly disentangle the learning of time-coherent fine-scale details from the 2D screen space embedding of humans.

Main Methods:

• The proposed method utilizes a two-convolutional neural network (CNN) architecture.
• The first CNN predicts a dynamic texture map with time-coherent high-frequency details based on pose information.
• The second CNN uses the output of the first CNN to condition the final video generation.

Main Results:

• The method demonstrates significant qualitative and quantitative improvements over state-of-the-art techniques.
• Applications include human reenactment and novel view synthesis from monocular video.
• Achieved enhanced temporal stability and preservation of fine-scale details like pose-dependent wrinkles.

Conclusions:

• The proposed approach effectively addresses artifacts in neural network-based human video synthesis.
• Explicitly disentangling fine details leads to more realistic and temporally coherent video generation.
• The method shows promise for advanced applications like human reenactment and novel view synthesis.