IIMCNet: Intra- and Inter-Modality Correlation Network for Hybrid EEG-fNIRS Brain-Computer Interface.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Hybrid Brain-Computer Interfaces (BCI) integrate multi-modality signals for enhanced accuracy and reliability.
• Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are portable and cost-effective for real-world BCI.
• Current BCI methods often neglect joint intra- and inter-modality feature extraction, limiting performance.

Purpose of the Study:

• To develop a novel network, IIMCNet, for effective joint feature extraction in hybrid BCIs.
• To integrate both inherent uni-modality features and cross-modality features from EEG and fNIRS signals.
• To improve the robustness and accuracy of BCI systems through comprehensive feature fusion.

Main Methods:

• Introduced the Intra- and Inter-modality Correlation Network (IIMCNet) for hybrid BCI.
• Employed a late fusion strategy (Intra-net) for uni-modality feature extraction (EEG, HbR, HbO).
• Utilized an early fusion strategy (Inter-net) with dilated convolution-based C-Nets for cross-modality feature extraction (EEG-HbR, EEG-HbO, HbR-HbO).
• Integrated intra-modality, inter-modality, and concatenated hybrid features into a deep supervision module.

Main Results:

• IIMCNet demonstrated superior performance compared to methods using only intra- or inter-modality correlations.
• The proposed IIMCNet significantly outperformed state-of-the-art methods in motor imagery tasks.
• IIMCNet also achieved superior results in mental arithmetic tasks, validating its effectiveness.

Conclusions:

• Joint extraction of intra- and inter-modality features is crucial for optimizing hybrid BCI performance.
• IIMCNet effectively captures complementary information from EEG and fNIRS signals.
• The IIMCNet approach offers a promising direction for advancing real-world BCI applications.