Memories are One-to-Many Mapping Alleviators in Talking Face Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Talking face generation aims to create realistic video portraits from audio input.
• The one-to-many mapping between audio and lip movements introduces ambiguity, leading to suboptimal visual results.
• Existing two-stage frameworks inadequately address this ambiguity due to missing information like emotions and facial details.

Purpose of the Study:

• To propose MemFace, a novel framework to enhance talking face generation.
• To address the ambiguity in audio-to-lip motion mapping by incorporating memory mechanisms.
• To improve the realism and detail of generated video portraits.

Main Methods:

• Introduced MemFace, a framework with implicit and explicit memory components.
• Utilized implicit memory in the audio-to-expression model for high-level semantic capture.
• Employed explicit memory in the neural-rendering model for synthesizing pixel-level details.

Main Results:

• MemFace consistently and significantly surpasses state-of-the-art results across various scenarios.
• The proposed memory mechanisms effectively complement missing information in the generation process.
• Improved visual quality and realism in generated talking face videos.

Conclusions:

• MemFace effectively resolves the one-to-many mapping problem in talking face generation.
• The integration of implicit and explicit memory significantly enhances the performance of audio-driven facial animation.
• MemFace represents a substantial advancement in generating realistic and detailed talking face videos.