Emotion Recognition Using Multi-View EEG-fNIRS and Cross-Attention Feature Fusion.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Artificial Intelligence

Background:

• Accurate emotion recognition is crucial for human-computer interaction and mental health.
• Existing methods often struggle with the complexity and subtlety of emotional states.
• Integrating multimodal neuroimaging data offers a promising avenue for improved accuracy.

Purpose of the Study:

• To propose a novel multimodal fusion module, FGCN-TCNN-CAF, for enhanced emotion recognition.
• To leverage differentiated modeling strategies for frequency, spatial, and temporal features from EEG and fNIRS signals.
• To validate the superiority of the multimodal approach over single-modal methods.

Main Methods:

• Utilized electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) for multimodal data acquisition.
• Employed a frequency-domain graph convolutional network (FGCN) for EEG frequency band analysis and a time-domain convolutional network (TCNN) for temporal and spatial feature extraction.
• Integrated a cross-attention fusion network (CAF) for interactive fusion of multimodal features.

Main Results:

• The proposed multi-view EEG approach demonstrated higher recognition accuracy than using EEG alone.
• Multimodal recognition results surpassed single-modal EEG by 1.73% and single-modal fNIRS by 6.65%.
• The FGCN-TCNN-CAF method achieved a state-of-the-art accuracy of 96.09% in emotion recognition.

Conclusions:

• The FGCN-TCNN-CAF module effectively fuses multimodal EEG-fNIRS data for superior emotion recognition.
• Differentiated modeling of signal features significantly improves performance.
• This multimodal approach represents a significant advancement in the field of affective computing.