PSPN: Pseudo-Siamese Pyramid Network for multimodal emotion analysis.

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Emotion recognition is crucial for healthcare and daily life.
• Electroencephalogram (EEG) is a key objective indicator for intense emotions.
• Current methods struggle with shallow/deep EEG features and single-modal limitations for complex emotions.

Purpose of the Study:

• To propose a novel multimodal emotion analysis framework integrating EEG and eye-related signals.
• To address limitations in current emotion recognition methods by incorporating complementary data sources.
• To enhance the accuracy and comprehensiveness of emotion recognition.

Main Methods:

• Developed a Pseudo-Siamese Pyramid Network (PSPN) for multimodal emotion analysis.
• Utilized Depthwise Separable Convolutional Pyramid (DSCP) for multi-level EEG feature extraction.
• Employed a fully connected subnetwork for eye-related signal feature extraction.
• Integrated a cross-modal dual-branch subnetwork for collaborative feature utilization.

Main Results:

• The PSPN model achieved high accuracy on public datasets (DEAP and SEED-IV).
• Achieved 96.02% accuracy for valence and 96.45% for arousal on the DEAP dataset.
• Reached 77.81% accuracy on the SEED-IV dataset.
• Demonstrated significant accuracy improvement through multimodal fusion compared to single-modal approaches.

Conclusions:

• Multimodal fusion using EEG and eye-related signals significantly enhances emotion recognition.
• The proposed PSPN model effectively integrates diverse features for robust emotion analysis.
• This approach offers a more comprehensive representation of complex human emotions.