EFS-NET: EEG-fNIRS multi-scale fusion network based on spatial calibration

Hybrid brain-computer interfaces (hBCIs) integrate multiple neuroimaging modalities and utilize their complementary information to address the inherent limitations of single-modality neural signal decoding. For electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) hybrid BCIs, advanced fusion algorithms are crucial to fully exploit the superior spatial localization capability of fNIRS and the millisecond-level temporal resolution of EEG. This work proposes an end-to-end spatial calibration-based multi-scale EEG-fNIRS fusion network named EFS-Net, which organically integrates EEG and fNIRS signals through a multi-scale spatio-temporal fusion architecture. The network consists of three complementary functional branches: a multi-scale temporal convolution branch for capturing rapidly changing cortical electrophysiological features of EEG, an EEG spatial branch for constructing latency-compensated cortical topographies to adapt to the delayed hemodynamic response of fNIRS, and a spatially calibrated fNIRS spatial branch for dynamically fusing spatial feature maps with EEG counterparts to generate temporally aligned and spatially enhanced neural representations. This three-branch fusion structure constructs abundant spatio-temporal feature embeddings and improves the discriminability of neural features. Evaluated on two public datasets including Word Generation (WG) and Mental Arithmetic (MA) with a rigorous subject-specific leave-one-session-out cross-validation protocol, EFS-Net achieves classification accuracies of 77.71 ± 8.23% and 81.69 ± 9.49% on the WG and MA datasets respectively, which surpasses state-of-the-art unimodal algorithms and traditional fusion models. Visualization results demonstrate that the designed alignment strategy can restore realistic cortical spatial distribution characteristics, providing a feasible solution for personalized neural signal decoding in hybrid BCIs.