Talking Face Generation With Audio-Deduced Emotional Landmarks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Computer Vision

Background:

• Image-based talking face generation synthesizes videos from facial images and audio.
• Existing methods often fail to synchronize emotions, leading to inaccurate facial expressions and poor image quality.

Purpose of the Study:

• To develop a high-quality talking face generation framework that synchronizes both lip movements and emotions with audio.
• To address limitations of current methods regarding emotion unsynchronization and image quality.

Main Methods:

• Proposed a two-stage framework: AMIGO (Audio Emotion-Aware Talking Face Generation).
• Stage one: A sequence-to-sequence (seq2seq) network generates emotional landmarks synchronized with audio, using coordinated visual emotion representation.
• Stage two: A feature-adaptive visual translation network refines landmarks into high-quality facial images using a feature-adaptive transformation module.

Main Results:

• The AMIGO framework generates talking face videos with cross-modally synced emotion.
• Significant improvements in image quality were achieved by fusing landmark and image representations.
• Extensive experiments on MEAD and CREMA-D datasets demonstrated superior performance compared to state-of-the-art methods.

Conclusions:

• The proposed AMIGO framework effectively generates high-quality, emotionally synchronized talking face videos.
• The novel approach addresses key limitations in current talking face generation techniques.
• The method shows strong potential for applications requiring realistic and emotionally expressive digital avatars.