Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Human emotions are complex, leading to overlapping features in recognition methods.
• Existing emotion recognition techniques struggle with accurate feature extraction and low precision.

Purpose of the Study:

• To propose an improved multimodal emotion recognition method using facial expressions and EEG signals.
• To enhance accuracy and overcome limitations of current emotion recognition systems.

Main Methods:

• Facial expression features were extracted using a bilinear convolution network (BCN).
• EEG signals were converted into frequency band image sequences and fused with expression features using BCN.
• A three-layer bidirectional LSTM with an attention mechanism was utilized for feature fusion and timing modeling.

Main Results:

• The proposed method effectively extracts emotion features from both expressions and EEG signals.
• The attention mechanism improved image feature representation.
• Experiments on MAHNOB-HCI and DEAP datasets demonstrated higher emotion recognition accuracy compared to existing methods.

Conclusions:

• The integration of facial expression and EEG data with a bidirectional LSTM and attention mechanism significantly improves emotion recognition accuracy.
• This multimodal approach offers a more effective way to understand and recognize human emotions.