Facial Emotion Recognition of 16 Distinct Emotions From Smartphone Videos: Comparative Study of Machine Learning and...

Area of Science:

• Computational psychology and affective computing.
• Development of artificial intelligence for mental health applications.

Background:

• Automatic emotion recognition (FER) is vital for advancing psychotherapeutic apps.
• Current models often overlook therapeutically relevant emotions beyond the basic six.
• Introduction of the novel Stress Reduction Training Through the Recognition of Emotions Wizard-of-Oz (STREs WoZ) dataset with 16 distinct emotions.

Purpose of the Study:

• To develop and assess deep learning-based FER models for binary and multiclass emotion classification.
• To compare the performance of automatic FER models against human observers.

Main Methods:

• Trained FER models on the STREs WoZ dataset (14,412 videos, 63 individuals, 16 emotions).
• Utilized appearance features (Facial Action Coding System via OpenFace) and deep-learned features (ResNet50).
• Employed recurrent neural network (RNN) architectures (RNN-convolution, RNN-attention, RNN-average) for classification.
• Validated models against 3 human observers on a test set of 3018 videos.

Main Results:

• FER models using appearance features outperformed deep-learned or combined features.
• The attention network with appearance features achieved the highest performance (92.9% UAR in binary, 59.0%-90.0% accuracy in multiclass).
• Human observers showed comparable performance in binary tasks (91.0% UAR) but superior accuracy in multiclass tasks (87.4%-99.8%).

Conclusions:

• This study provides a foundational step for enhancing emotion-focused psychotherapeutic interventions via smartphone apps.
• Further research is required to improve automatic FER model performance for practical clinical applications.