Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Motor imagery-based brain-computer interfaces (MI-BCI) translate brain signals into commands for external devices.
• Multimodal MI-BCI, integrating diverse neural signals like EEG and fNIRS, offers potential for enhanced decoding accuracy due to complementary information.
• Signal heterogeneity across modalities presents a significant challenge for effective multimodal decoding.

Purpose of the Study:

• To develop and evaluate a novel multimodal decoding neural network for motor imagery-based brain-computer interfaces (MI-BCI).
• To address the challenge of signal heterogeneity in multimodal MI-BCI by designing effective feature alignment and fusion strategies.
• To improve the decoding accuracy of MI-BCI by leveraging complementary information from electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) signals.

Main Methods:

• A multimodal neural network architecture was proposed for MI decoding.
• Spatial feature alignment losses were introduced to enhance feature representations from heterogeneous EEG and fNIRS data.
• An attention-based modality fusion module was employed for temporal feature alignment and fusion.

Main Results:

• The proposed decoding method demonstrated superior accuracy compared to existing methods on both self-collected and public datasets.
• Ablation studies confirmed the effectiveness of individual components of the proposed multimodal decoding approach.
• Feature distribution analysis indicated that the proposed alignment losses improved the feature representations for both EEG and fNIRS modalities.

Conclusions:

• The developed multimodal decoding neural network effectively integrates EEG and fNIRS signals for improved motor imagery decoding.
• The proposed spatial feature alignment and attention-based fusion mechanisms are crucial for handling signal heterogeneity.
• This approach shows significant potential for advancing the performance of MI-BCI systems.