Disentangled Multimodal Spatiotemporal Learning for Hybrid EEG-fNIRS Brain-Computer Interface.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Hybrid Brain-computer interfaces (BCI) integrate electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) for comprehensive brain activity detection, leveraging EEG's temporal and fNIRS's spatial resolution.
• Current integration methods struggle to capture spatiotemporal coupling features and inter-modality correlations, often resulting in unrefined multimodal representations.
• Holistic learning paradigms in existing methods lead to redundant feature extraction, limiting the discriminative power of hybrid BCI systems.

Purpose of the Study:

• To propose a novel Disentangled Multimodal Spatiotemporal Learning (DMSL) method for hybrid EEG-fNIRS BCI systems.
• To enhance the extraction of spatiotemporal coupling features and inter-modality correlations between EEG and fNIRS signals.
• To develop a unified framework for disentangled representation learning and multimodal spatiotemporal coupling.

Main Methods:

• DMSL employs a compact convolutional module with 1D temporal and spatial convolutions to extract spatiotemporal patterns from individual modalities.
• A multimodal attention interaction module captures inter-modality correlations, refining modality-specific representations.
• An adaptive multi-branch graph convolutional module, utilizing reconstructed channels and modality constraints, disentangles common and specific representations for effective fusion and task prediction.

Main Results:

• The proposed DMSL method achieved state-of-the-art performance on mental arithmetic, motor imagery, and emotion recognition tasks.
• DMSL outperformed existing methods by 2.34% in mental arithmetic, 0.59% in motor imagery, and 1.47% in emotion recognition.
• The results validate the effectiveness of DMSL in improving EEG-fNIRS decoding accuracy and demonstrating strong generalization capabilities.

Conclusions:

• The DMSL method significantly enhances the decoding performance of hybrid EEG-fNIRS BCI systems.
• The disentangled representation learning approach effectively captures crucial spatiotemporal coupling features and inter-modality correlations.
• DMSL shows promising potential for advancing BCI applications requiring robust and accurate brain activity interpretation.