Multimodal physiological signal emotion recognition based on multi-head cross attention with representation learning.

Area of Science:

• Affective Computing
• Biomedical Signal Processing
• Machine Learning for Emotion Recognition

Background:

• Physiological signals offer objective emotion recognition, but current multimodal fusion methods struggle with complex interactions.
• Existing techniques often fail to fully leverage complementary information between different physiological signals.
• This limits the comprehensive understanding and accurate classification of affective states.

Purpose of the Study:

• To propose a novel framework for multimodal physiological emotion recognition.
• To effectively learn and extract features from multiple modalities simultaneously, mimicking human emotion perception.
• To enhance the accuracy and robustness of emotion recognition systems.

Main Methods:

• A dual-branch representation learning architecture processes electroencephalography (EEG) and peripheral signals separately.
• A tailored cross-attention mechanism is employed for multimodal signal fusion.
• The framework aims to comprehensively exploit complementary information and improve feature learning.

Main Results:

• The proposed model achieved superior performance in emotion classification tasks on public datasets (DEAP, SEED-IV).
• Experimental results validate the model's ability to effectively extract and fuse features from multimodal physiological signals.
• The framework demonstrated enhanced accuracy and robustness compared to state-of-the-art models.

Conclusions:

• The novel framework significantly advances multimodal physiological emotion recognition.
• The model's effectiveness in feature extraction and fusion holds promise for affective computing.
• This research has implications for healthcare and human-computer interaction applications.