MFA-CNN: An Emotion Recognition Network Integrating 1D-2D Convolutional Neural Network and Cross-Modal Causal...

Background/Objectives: It has become a major direction of research in affective computing to explore the brain-information-processing mechanisms based on physiological signals such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). However, existing research has mostly focused on feature- and decision-level fusion, with little investigation into the causal relationship between these two modalities. Methods: In this paper, we propose a novel emotion recognition framework for the simultaneous acquisition of EEG and fNIRS signals. This framework integrates the Granger causality (GC) method and a modality-frequency attention mechanism within a convolutional neural network backbone (MFA-CNN). First, we employed GC to quantify the causal relationships between the EEG and fNIRS signals. This revealed emotional-processing mechanisms from the perspectives of neuro-electrical activity and hemodynamic interactions. Then, we designed a 1D2D-CNN framework that fuses temporal and spatial representations and introduced the MFA module to dynamically allocate weights across modalities and frequency bands. Results: Experimental results demonstrated that the proposed method outperforms strong baselines under both single-modal and multi-modal conditions, showing the effectiveness of causal features in emotion recognition. Conclusions: These findings indicate that combining GC-based cross-modal causal features with modality-frequency attention improves EEG-fNIRS-based emotion recognition and provides a more physiologically interpretable view of emotion-related brain activity.