Comparing supervised and unsupervised approaches to multimodal emotion recognition.

Area of Science:

• Affective computing
• Human-computer interaction
• Multimodal emotion recognition

Background:

• Emotion classification is crucial for human-computer interaction.
• Integrating vocal and facial cues enhances recognition accuracy.
• Understanding emotion expression organization aids AI development.

Purpose of the Study:

• To classify 18 emotions from video recordings using multimodal data.
• To investigate supervised and unsupervised methods for emotion recognition.
• To identify vocal and facial feature patterns associated with specific emotions.

Main Methods:

• Supervised classification using vocal (acoustic parameters) and facial (action units) features.
• Late fusion approach combining unimodal classifier outputs (Elastic Net, Random Forest).
• Exploratory unsupervised classification with clustering and dimensionality reduction.

Main Results:

• The best performance (AUC = 0.88) was achieved by merging vocal and facial classifiers.
• All 18 emotions were recognized above chance, with varying accuracy (e.g., high for amusement, anger; low for shame).
• Unsupervised methods revealed patterns related to valence, arousal, actor, and gender, but not distinct emotion categories.

Conclusions:

• Multimodal fusion significantly improves emotion classification accuracy.
• Vocal and facial features provide complementary information for emotion recognition.
• Unsupervised pattern identification offers insights into emotion expression organization and aids complex classification tasks.