Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Emotion recognition is crucial for human-machine interaction.
• Existing methods face challenges in multi-modal integration and real-time processing.
• Deep learning models require significant computational power, impacting robustness.

Purpose of the Study:

• To propose Deep-Emotion, a deep learning-based multimodal emotion recognition (MER) system.
• To adaptively integrate discriminating features from facial expressions, speech, and EEG.
• To improve the performance, real-time detection, and robustness of emotion recognition.

Main Methods:

• Developed a three-branch framework: improved GhostNet for facial expressions, lightweight fully convolutional neural network (LFCNN) for speech, and tree-like LSTM (tLSTM) for EEG.
• Employed decision-level fusion to integrate results from the three modalities.
• Utilized improved GhostNet to alleviate overfitting and enhance classification accuracy.

Main Results:

• Extensive experiments on CK+, EMO-DB, and MAHNOB-HCI datasets validated the Deep-Emotion method.
• Demonstrated the advanced nature and superiority of the proposed MER approach.
• Achieved comprehensive and accurate emotion recognition through multi-modal fusion.

Conclusions:

• The Deep-Emotion method is advanced and effective for multimodal emotion recognition.
• The proposed approach is feasible and superior to existing methods.
• Deep-Emotion enhances human-machine interaction through accurate and robust emotion detection.