Live Speech Driven Head-and-Eye Motion Generators.

Area of Science:

• Computer Graphics
• Human-Computer Interaction
• Biomedical Engineering

Background:

• Realistic animation of human facial motion, including head, eye gaze, and eyelid movements, is crucial for virtual characters and human-computer interaction.
• Existing methods often struggle to generate these motions simultaneously and realistically from speech input.

Purpose of the Study:

• To develop a fully automated framework for generating simultaneous and realistic head motion, eye gaze, and eyelid motion driven by live or recorded speech.
• To introduce a novel data acquisition technique for capturing high-fidelity facial motion data.

Main Methods:

• Utilizing separate yet interrelated statistical models: Gaussian Mixture Models for head motion, Nonlinear Dynamic Canonical Correlation Analysis for eye gaze, and nonnegative linear regression with log-normal distribution for eyelid motion.
• Employing a mocap+video hybrid data acquisition technique for simultaneous recording of high-fidelity head and eye movements.

Main Results:

• User studies employing paired comparison methodology demonstrated that the proposed framework significantly outperforms state-of-the-art algorithms for head and eye motion generation.
• The framework successfully generates realistic and synchronized head, eye gaze, and eyelid motions from speech input.

Conclusions:

• The developed automated framework offers a significant advancement in generating realistic, speech-driven facial animations.
• The novel data acquisition technique provides high-fidelity data essential for training such generative models.