A New Dual-Attention Multi-Node Fusion Network for EEG-fNIRS Motor Imagery Classification

Brain-computer interface (BCI) based on motor imagery (MI) can realize the direct control of external devices by decoding different signals. The decoding of MI based on electroencephalogram (EEG) suffers from low spatial resolution and is susceptible to noise. Functional near-infrared spectroscopy (fNIRS) has attracted increasing attention as a complementary modality. There have been attempts to fuse the two types of signals, but their spatio-temporal characteristics have not been fully explored. We propose a new multimodal EEG-fNIRS fusion MI classification and recognition model based on a dual attention mechanism. The model comprises two feature extraction branches and a central fusion network. We set two fusion layers in the central fusion network to exploit the spatio-temporal features of EEG and fNIRS. To reduce redundancy and mine correlation characteristics of multiple sensors, the features are fused in the filter dimension to prevent adverse effects between signals during fusion, thereby enabling the deep network to learn cross modal correlations while reducing mutual interference. The method is evaluated on two multimodal datasets. Experiments show that DAMFNet outperforms STA-Net and M2NN by 4.49% and 2.88% on Dataset1, respectively, and shows competitive performance on Dataset2. The code is available at https://github.com/useflf/DAMFNet.