BiGSTF-Net: inter-modal mutual guidance and intra-modal spatio-temporal fusion for EEG-fNIRS cognitive classification

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offer complementary data for brain-computer interfaces (BCIs).
• Integrating these heterogeneous signals (electrophysiological and hemodynamic) presents significant challenges.
• Effective fusion is crucial for advancing BCI performance.

Purpose of the Study:

• To develop a multimodal architecture, BiGSTF-Net, for enhanced cognitive state decoding.
• To leverage the complementary temporal and spatial information from EEG and fNIRS signals.
• To address the challenges posed by the heterogeneous nature of multimodal neural signals.

Main Methods:

• Proposed BiGSTF-Net architecture utilizing heterogeneous spatio-temporal feature extractors.
• Introduced Modal Residual Interaction Unit (MRIU) for bidirectional cross-modal guidance.
• Employed Spatio-Temporal Gated Unit (STGU) for intra-modal feature integration.

Main Results:

• BiGSTF-Net demonstrated superior performance in cognitive state decoding compared to existing multimodal fusion methods.
• Cross-session evaluations on multiple BCI datasets confirmed consistent outperformance.
• Ablation studies validated the efficacy of individual architectural components.

Conclusions:

• The proposed BiGSTF-Net framework offers an effective solution for multimodal neural signal decoding.
• The architecture successfully integrates complementary EEG and fNIRS data.
• This approach advances the capabilities of brain-computer interfaces.