Decoding Hybrid EEG-fNIRS Upper Limb Motor Execution with Capsule Dynamic Graph Convolutional Neural Network.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Machine Learning

Background:

• Decoding motor execution is crucial for brain-computer interfaces (BCIs).
• Combining electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) offers complementary information for enhanced BCI performance.
• Existing methods often struggle with effective multimodal fusion.

Purpose of the Study:

• To propose and evaluate a novel capsule dynamic graph convolution network (EF-CapsDGCN) for accurate decoding of upper limb motor execution.
• To investigate the efficacy of fusing EEG and fNIRS signals using the proposed EF-CapsDGCN model.
• To compare the performance of EF-CapsDGCN against state-of-the-art methods and single-modality approaches.

Main Methods:

• Feature extraction from EEG and fNIRS signals using a shared convolutional architecture.
• Dynamic routing of features to capsules and formation of multimodal capsules.
• Learning hidden representations via dynamic graph convolution on capsule nodes.
• Multi-head self-attention mechanism for feature integration before classification.

Main Results:

• The proposed EF-CapsDGCN achieved superior classification performance on the HYGRIP multimodal dataset.
• Multimodal EEG-fNIRS fusion using EF-CapsDGCN resulted in at least an 8% increase in classification accuracy compared to single modalities.
• The model demonstrated significant improvements over existing methods like ANN, DeepConvNet, DNN, and EF-Net.

Conclusions:

• Capsule dynamic graph convolution is effective for multimodal fusion of EEG and fNIRS signals.
• The EF-CapsDGCN model shows great promise for accurate motor execution decoding in EEG-fNIRS-based BCIs.
• This study provides an effective solution for multimodal BCI decoding, with clinical relevance for motor impairment rehabilitation.