Diverse Code Query Learning for Speech-Driven Facial Animation.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Speech-driven facial animation synthesizes realistic 3D talking faces from audio.
• Existing methods often lack diversity and struggle with stochastic facial motion.
• Generative models face challenges in mode coverage on small datasets.

Purpose of the Study:

• To develop a method for diverse speech-driven facial animation synthesis.
• To address the challenge of generating varied facial motions from a single audio signal.
• To enable controllable and diverse facial animation generation.

Main Methods:

• Proposing a model that predicts multiple animation samples conditioned on the same audio.
• Utilizing a diversity-promoting loss to explore the facial latent space.
• Employing a vector-quantized variational auto-encoding mechanism for facial priors.
• Sequentially predicting and composing different facial portions for full-face animation.

Main Results:

• Achieved state-of-the-art performance in quantitative and qualitative evaluations.
• Demonstrated superior sample diversity compared to prior methods.
• Successfully generated diverse yet plausible, speech-faithful facial motions.
• Enabled controllable generation of different facial parts.

Conclusions:

• The proposed unified framework successfully synthesizes diverse and controllable facial animations.
• The method effectively addresses the limitations of existing deterministic and less diverse approaches.
• This work advances the field of speech-driven facial animation by incorporating stochasticity and diversity.