Multimodal Emotion Recognition from EEG and ECG Signals via Parallel Fusion of Graph Convolution and LSTMs

Area of Science:

• Neuroscience
• Machine Learning
• Signal Processing

Background:

• Emotion recognition is crucial for human-computer interaction.
• Combining electroencephalogram (EEG) and electrocardiogram (ECG) signals improves emotion recognition accuracy.
• Previous methods often neglect EEG functional connectivity in multimodal fusion.

Purpose of the Study:

• To develop an advanced emotion recognition model by integrating EEG functional connectivity and ECG temporal features.
• To leverage dynamical graph convolution and self-attention mechanisms for enhanced multimodal fusion.
• To establish a robust baseline for future research in EEG and ECG-based emotion recognition.

Main Methods:

• Utilized dynamical graph convolution to learn EEG functional connectivity.
• Employed Long short-term memory (LSTM) networks for ECG temporal feature extraction.
• Implemented a parallel fusion strategy with self-attention for synergistic cross-modal feature integration.
• Evaluated the proposed Parallel fusion of Graph Convolution and Long short-term memory (PGCL) model on the DREAMER dataset.

Main Results:

• The PGCL model achieved superior performance compared to state-of-the-art methods in subject-dependent emotion recognition.
• Demonstrated significant improvements in mean classification accuracy for low/high valence (10.41%) and arousal (9.54%) predictions.
• The integration of EEG functional connectivity proved effective in enhancing multimodal emotion recognition.

Conclusions:

• The proposed PGCL model offers a powerful new baseline for multimodal EEG and ECG emotion recognition.
• Dynamical graph convolution for EEG functional connectivity and parallel fusion enhance recognition accuracy.
• This research provides valuable insights into physiological signal-based emotion recognition.