DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation.

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Current talking head generation methods primarily use 2D facial information, limiting 3D accuracy.
• Dense 3D facial geometry, like pixel-wise depth, is crucial for accurate 3D reconstruction and noise suppression.
• Acquiring dense 3D annotations for facial videos is expensive and challenging.

Purpose of the Study:

• To develop a self-supervised method for learning dense 3D facial geometry (depth) from videos without 3D annotations.
• To improve the accuracy of facial keypoint estimation for motion field generation.
• To create a 3D-aware attention mechanism for enhanced facial geometry capture in talking head synthesis.

Main Methods:

• A novel self-supervised approach to learn dense 3D facial geometry (depth) from face videos, eliminating the need for camera parameters or 3D annotations.
• A strategy for learning pixel-level uncertainties to identify reliable pixels for geometry learning.
• A geometry-guided facial keypoint estimation module for accurate motion field generation.
• A 3D-aware cross-modal attention mechanism integrating appearance and depth for coarse-to-fine facial geometry capture.

Main Results:

• The proposed framework successfully generates highly realistic reenacted talking videos.
• New state-of-the-art performances were achieved on the VoxCeleb1, VoxCeleb2, and HDTF benchmarks.
• The method effectively utilizes self-supervised learning for dense 3D facial geometry extraction.

Conclusions:

• The developed framework offers a cost-effective and efficient solution for 3D talking head generation.
• Self-supervised learning of 3D facial geometry significantly enhances the realism and accuracy of generated videos.
• The integration of geometry-guided keypoints and cross-modal attention advances the state-of-the-art in talking head synthesis.