CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset

Area of Science:

• Psychology
• Computer Science
• Human-Computer Interaction

Background:

• Human emotional states are expressed through facial and vocal cues.
• Understanding multi-modal emotion perception is crucial for human-computer interaction and affective computing.

Purpose of the Study:

• To introduce a comprehensive audio-visual dataset for studying multi-modal emotion expression and perception.
• To analyze the accuracy of emotion recognition across audio-only, visual-only, and audio-visual modalities.

Main Methods:

• Collected 7,442 audio-visual clips of 91 actors displaying basic emotions (happy, sad, anger, fear, disgust, neutral).
• Utilized crowd-sourcing with 2,443 raters to collect categorical emotion labels and intensity values.
• Evaluated human recognition rates for audio, visual, and combined audio-visual perception.

Main Results:

• Audio-visual emotion recognition achieved 63.6% accuracy, outperforming audio-only (40.9%) and visual-only (58.2%).
• Recognition accuracy varied by emotion, with neutral and happy emotions being most accurately recognized.
• Disgust and fear recognition benefited most from combined audio-visual cues, while anger and happiness were recognizable from single modalities.

Conclusions:

• The presented dataset is a valuable resource for advancing research in multi-modal emotion recognition.
• Simultaneous audio-visual information enhances the perception of certain emotions more than others.
• Findings have implications for developing more emotionally intelligent AI systems.