An Assessment of In-the-Wild Datasets for Multimodal Emotion Recognition.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Affective Computing

Background:

• Multimodal emotion recognition utilizes diverse data (faces, speech, text) for identifying human emotions.
• Current deep learning models often struggle in real-world conditions due to training on controlled datasets.

Purpose of the Study:

• To assess the strengths and weaknesses of in-the-wild datasets for multimodal emotion recognition.
• To evaluate the suitability of existing datasets for practical emotion AI applications.

Main Methods:

• Evaluation of four in-the-wild datasets: AFEW, SFEW, MELD, and AffWild2.
• Utilized a pre-designed multimodal architecture for performance assessment.
• Measured performance using accuracy and F1-Score metrics.

Main Results:

• In-the-wild datasets exhibit limitations for standalone multimodal emotion recognition.
• Dataset suitability is often constrained by their original design purpose (e.g., face or speech recognition).

Conclusions:

• Individual in-the-wild datasets are insufficient for robust multimodal emotion recognition.
• Combining multiple datasets and balancing class samples is crucial for improved real-world performance.