Multimodal Emotion Recognition from EEG and ECG Signals via Parallel Fusion of Graph Convolution and LSTMs

Emotion recognition has become an active research area due to its wide applications. While various physiological signals contribute to the regulation of human emotions, previous studies have shown that integrating electroencephalogram (EEG) and electrocardiogram (ECG) features yields superior performance in emotion recognition. However, previous studies for multimodal fusion of EEG and ECG features overlooked the effectiveness of EEG functional connectivity. For further performance improvements, we leverage dynamical graph convolution, which learns EEG functional connectivity to interplay with ECG temporal features for emotion recognition. Moreover, these separately learned multimodal EEG and ECG features are stacked and the following self-attention induces synergistic effects from cross-modal EEG and ECG features. Subsequently, these output features are concatenated and downsampled for final classification. We evaluated our proposed model, namely Parallel fusion of Graph Convolution and Long short-term memory (PGCL), on DREAMER, which is a popular dataset including EEG and ECG signals for emotion recognition. In the subject-dependent setting, we compared the PGCL's performance with two state-of-the-art methods related to our study, and the proposed method outperformed the stronger baseline model by 10.41% and 9.54% for low/high valence and arousal predictions in mean classification accuracy. This result signifies that our proposed method can give more insights and be used as a powerful baseline model for future works on multimodal EEG and ECG emotion recognition.Clinical relevance- This paper learns multimodal physiological feature representations of EEG functional connectivity and ECG temporal sequences via parallel fusion of graph convolution and LSTMs, which contributes to improved emotion recognition performance.