Emotion recognition based on the temporal patterns of electroencephalogram signals and electrodermal response signals...

Emotion recognition using physiological signals plays an important role in affective neuroscience and human-centered artificial intelligence. Current methods still face challenges in long-range temporal dependency modeling and explicit central-autonomic coupling representation, while generalization under subject-independent protocols needs further improvement.

This study proposes a Transformer-based multimodal framework for four-class discrete emotion recognition (neutral, happiness, sadness, and fear) by jointly modeling EEG and GSR signals. The architecture integrates temporal self-attention and bidirectional cross-modal attention. Experiments were conducted on 42 neurologically healthy adults with a controlled audiovisual emotion elicitation paradigm, evaluated using subject-independent five-fold cross-validation.

The model achieved a mean classification accuracy of 87.42% ± 2.13%, with precision of 87.6%, recall of 87.4%, and F1-score of 87.5%. It outperformed CNN and Bi-LSTM baselines by 4.91% and 6.38%, respectively. Multimodal fusion significantly boosted high-arousal emotion recognition, with fear accuracy increasing from 82.11% (EEG-only) to 88.63% (p = 0.004).

These findings confirm that long-range temporal modeling and explicit cross-modal interaction can substantially improve multimodal physiological emotion recognition. The proposed framework is scalable and interpretable, advances central-autonomic coupling modeling, enhances generalization via strict subject-independent validation, and supports physiological interpretability through attention visualization and modality sensitivity analysis.