MFA-CNN: An Emotion Recognition Network Integrating 1D-2D Convolutional Neural Network and Cross-Modal Causal...

Area of Science:

• Affective computing
• Neuroscience
• Biomedical engineering

Background:

• Physiological signals like EEG and fNIRS are key to understanding brain mechanisms in affective computing.
• Current research often overlooks the causal links between EEG and fNIRS signals, focusing instead on feature or decision-level fusion.

Purpose of the Study:

• To develop a novel emotion recognition framework using simultaneous EEG and fNIRS acquisition.
• To investigate the causal relationships between EEG and fNIRS signals for enhanced emotion recognition.

Main Methods:

• Proposed a framework integrating Granger causality (GC) with a modality-frequency attention mechanism (MFA) within a convolutional neural network (CNN).
• Quantified causal relationships between EEG and fNIRS signals using GC to understand neuro-electrical and hemodynamic interactions.
• Developed a 1D2D-CNN framework with an MFA module for fusing temporal, spatial, and cross-modal information.

Main Results:

• The proposed method significantly outperformed existing baselines in emotion recognition tasks.
• Demonstrated the effectiveness of incorporating causal features derived from EEG-fNIRS interactions.
• Achieved superior performance in both single-modal and multi-modal recognition scenarios.

Conclusions:

• Combining GC-based cross-modal causal features with modality-frequency attention improves EEG-fNIRS emotion recognition.
• The framework offers a more physiologically interpretable approach to understanding emotion-related brain activity.
• Highlights the importance of exploring causal relationships for advancing affective computing.