Related Experiment Videos
Multimodal emotion recognition using hybrid deep feature fusion under speaker-independent evaluation
Elhossiny Ibrahim1, Mohamed Ezzat Ghoraba2, Ahmed Ezzat Ghoraba3
1Department of Computer Science and Engineering, Faculty of Electronic Engineering, Menoufia University, Menouf, Egypt. Elhossiny@el-eng.menofia.edu.eg.
Scientific Reports
|June 24, 2026
Summary
This study presents a novel multimodal system for emotion recognition using voice and facial expressions. The advanced fusion technique achieves high accuracy, improving AI
Area of Science:
- Artificial Intelligence
- Human-Computer Interaction
- Machine Learning
Background:
- Emotion recognition is a complex challenge for AI systems, hindering human-centric perception.
- Current AI agents often struggle with interpreting human emotions from multimodal cues.
Purpose of the Study:
- To develop a novel multimodal emotion recognition system integrating voice and facial expressions.
- To enhance machine understanding of human emotions through complementary sensory channels.
Main Methods:
- Utilized a multimodal deep feature fusion technique combining handcrafted audio (MFCCs) and deep visual features (VGGFace).
- Employed a hybrid fusion strategy (concatenation, cross-attention, gated, multiplicative) for feature integration.
- Evaluated on RAVDESS and CREMA-D datasets under random-split and speaker-independent protocols.
Main Results:
- Achieved 95.83% accuracy on RAVDESS (random split) and 73.54% on CREMA-D (random split).
- Demonstrated speaker-independent accuracy of 48.06% (LOSO) on RAVDESS and 53.12% (5-fold CV) on CREMA-D.
- The hybrid fusion effectively captured cross-modal interactions for robust emotion recognition.
Conclusions:
- The proposed multimodal system shows significant potential for accurate emotion recognition.
- Hybrid fusion strategies are effective in integrating diverse feature types for improved performance.
- Speaker-independent evaluation highlights challenges and areas for future improvement in real-world applications.
Related Concept Videos
01:20Labeling Emotion
Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
01:24Facial Feedback Hypothesis
Charles Darwin proposed that facial expressions are an evolutionary adaptation for communication. He argued that these expressions are not influenced by culture but are universal across species. For example, a snarling expression with exposed teeth signals a threat in many animals, including humans. Darwin also suggested that displaying an emotion can intensify the feeling. Smiling, for example, could enhance one's sense of happiness. This idea laid the foundation for understanding the role of...