MSDSANet: Multimodal Emotion Recognition Based on Multi-Stream Network and Dual-Scale Attention Network Feature...

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Current electroencephalography (EEG) emotion recognition models face limitations in feature representation granularity and spatiotemporal dependence modeling.
• Effective emotion recognition requires capturing complex patterns in brain and eye activity.

Purpose of the Study:

• To propose a novel multimodal emotion recognition model that addresses the shortcomings of existing EEG-based approaches.
• To enhance feature representation and spatiotemporal modeling for improved emotion recognition accuracy.

Main Methods:

• A multimodal model integrating multi-scale feature representation and attention mechanisms was developed.
• The model employs a multi-stream network for shallow EEG feature extraction and a dual-scale attention module for shallow electrooculography (EOG) feature extraction.
• Multi-scale and multi-granularity feature fusion was utilized to improve feature richness and discriminability.

Main Results:

• The proposed model demonstrated superior performance compared to existing models on two benchmark datasets.
• Enhanced feature fusion led to richer and more discriminative multimodal feature representations.
• The integration of attention mechanisms improved the model's ability to capture spatiotemporal dependencies.

Conclusions:

• The developed multimodal emotion recognition model effectively overcomes limitations in EEG feature representation and spatiotemporal modeling.
• The proposed approach offers a significant advancement in the field of affective computing and emotion recognition.
• This model shows promise for real-world applications requiring accurate and robust emotion detection.