Expression-EEG Bimodal Fusion Emotion Recognition Method Based on Deep Learning.

Area of Science:

• * Affective computing and human-computer interaction.
• * Deep learning and signal processing for biometric recognition.

Background:

• * Single biometric recognition methods for emotion classification suffer from low accuracy in real-world scenarios.
• * Existing emotion recognition techniques often face limitations due to individual modality constraints.

Purpose of the Study:

• * To propose a novel deep learning-based bimodal fusion method for emotion recognition.
• * To enhance the accuracy and efficiency of emotion classification by integrating facial expression and EEG data.
• * To address the limitations of unimodal emotion recognition systems.

Main Methods:

• * Utilized an improved VGG-FACE network for rapid facial expression feature extraction.
• * Employed the wavelet soft threshold algorithm for artifact removal and feature extraction from EEG signals.
• * Implemented a decision fusion strategy combined with long and short-term memory (LSTM) networks for bimodal fusion and classification.

Main Results:

• * Achieved a high emotion recognition accuracy of 0.89 on the MAHNOB-HCI dataset.
• * Demonstrated an accuracy improvement of 8.51% compared to traditional LSTM models.
• * Reduced the identification method's running time by approximately 20 seconds.

Conclusions:

• * The proposed deep learning-based bimodal fusion method significantly improves emotion recognition accuracy.
• * Integrating facial expression and EEG data offers a more robust approach to emotion classification.
• * The method provides a computationally efficient and accurate solution for emotion recognition applications.