STeCANet: spatio-temporal cross attention network for brain computer interface systems using EEG-fNIRS signals.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Machine Learning

Background:

• Multimodal neuroimaging fusion enhances brain-computer interface (BCI) performance by integrating complementary neural dynamics.
• Existing fusion frameworks struggle with temporal asynchrony and adaptive integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), limiting generalization.

Purpose of the Study:

• Develop an adaptive fusion framework to align and integrate EEG and fNIRS representations.
• Improve cross-session and cross-subject generalization in BCI applications.

Main Methods:

• Propose STeCANet, a Spatiotemporal Cross-Attention Network for hierarchical attention-based alignment of EEG and fNIRS signals.
• Utilize fNIRS-guided spatial attention, EEG-fNIRS temporal alignment, adaptive fusion, and adversarial training for robust cross-modal interaction and spatiotemporal consistency.

Main Results:

• STeCANet significantly outperforms unimodal and multimodal baselines in session-independent and subject-independent settings across motor imagery, mental arithmetic, and word generation tasks.
• Ablation studies confirm the effectiveness of individual sub-modules and loss functions, including domain adaptation, in enhancing classification accuracy and robustness.

Conclusions:

• STeCANet provides a robust and interpretable solution for advanced BCI applications.
• The proposed adaptive fusion framework effectively addresses limitations in current multimodal neuroimaging integration for BCIs.