Time-Delay Neural Network for Continuous Emotional Dimension Prediction From Facial Expression Sequences

Area of Science:

• Computer Science
• Artificial Intelligence
• Psychology

Background:

• Automatic continuous affective state prediction from naturalistic facial expressions is crucial for human-computer interaction.
• Modeling the temporal dynamics of naturalistic expressions presents a significant research challenge.

Purpose of the Study:

• To propose a novel two-stage automatic system for continuous affective dimension value prediction from facial expression videos.
• To effectively model temporal relationships between consecutive predictions for improved emotion recognition.

Main Methods:

• A two-stage system combining traditional regression for frame-level classification and a time-delay neural network (TDNN) for temporal modeling.
• The TDNN models temporal relationships, leveraging previously classified frames to capture slow-changing emotional dynamics.
• The system was evaluated on three distinct facial expression video datasets.

Main Results:

• The proposed two-stage approach significantly enhances the performance of continuous emotional state estimation.
• The TDNN effectively models temporal information, overcoming biases from high frame-to-frame feature variability.
• The system achieved top performance in the affect recognition sub-challenge of the Third International Audio/Visual Emotion Recognition Challenge.

Conclusions:

• The novel two-stage system with TDNN integration offers a robust solution for continuous affective state prediction.
• Separating dynamics modeling from feature-based prediction allows for more effective exploitation of temporal information.
• This approach represents a significant advancement in recognizing naturalistic facial expressions for HCI applications.