Development of an Interactive Digital Human with Context-Sensitive Facial Expressions

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Conventional digital human systems struggle with complex emotions and real-time responsiveness.
• Limitations include multi-emotion co-occurrence, dynamic expression, and semantic accuracy.

Purpose of the Study:

• To propose a digital human system framework integrating multimodal emotion recognition and compound facial expression generation.
• To establish a real-time pipeline for compound emotional expression via "speech semantic parsing-multimodal emotion recognition-Action Unit (AU)-level 3D facial expression control."

Main Methods:

• Utilized ResNet18 for emotion classification on the AffectNet dataset.
• Developed an Action Unit (AU) motion curve driving module on Unreal Engine with a state-machine for dynamic emotion synthesis.
• Employed Generative Pre-trained Transformer (GPT) for semantic analysis and generating language-driven facial responses.

Main Results:

• Demonstrated significant improvements in facial animation quality.
• Increased naturalness from 3.54 to 3.94 and semantic congruence from 3.44 to 3.80.
• Validated the system's capability for realistic and emotionally coherent real-time expression generation.

Conclusions:

• The proposed system offers a complete technical framework for high-fidelity digital humans with affective interaction.
• Provides a practical foundation for advanced digital human development.
• Successfully addresses limitations in current digital human facial expression systems.