A Comprehensive Review of Multimodal Emotion Recognition: Techniques, Challenges, and Future Directions

Area of Science:

• Computer Science
• Artificial Intelligence
• Biomedical Engineering

Background:

• Human emotion is complex, often conveyed through multiple sensory channels.
• Unimodal emotion recognition systems have limitations in capturing nuanced affective states.
• Biomimetics offers a framework for developing advanced sensing systems inspired by biological principles.

Purpose of the Study:

• To provide a comprehensive review of multimodal emotion recognition (MER).
• To frame MER as a bio-inspired sensing paradigm.
• To highlight advancements, challenges, and future directions in MER.

Main Methods:

• Review of existing literature on MER systems.
• Analysis of MER system structures, feature extraction, and fusion strategies.
• Identification of key challenges and future research avenues.

Main Results:

• MER systems integrate speech, visual, and text data for richer emotion analysis.
• Biomimetics provides a foundation for MER by emulating human multisensory fusion.
• Key advancements and milestones in MER have been identified.

Conclusions:

• MER offers more robust emotion recognition than unimodal approaches.
• Future MER research should focus on lightweight models, cross-corpus generalizability, and incorporating additional modalities.
• Improving accuracy, explainability, and practicality is crucial for real-world MER applications.