Context-aware Multimodal Auditory BCI Classification through Graph Neural Networks.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Machine Learning

Background:

• Brain-computer interface (BCI) systems often overlook the integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) with participant topological information.
• Multimodality analysis combining EEG and fNIRS to enhance BCI performance is underexplored.

Purpose of the Study:

• To present a multimodal data fusion framework for exploiting synergistic properties in neural signals.
• To develop a context-aware graph neural network (GNN) model for improved auditory task classification using inter-subject relationships.

Main Methods:

• A multimodal data fusion framework was developed to integrate EEG and fNIRS signals.
• A context-aware GNN model was designed, treating auditory oddball task trials as context-aware nodes and utilizing pairwise relationships among participants.
• Experiments involved auditory oddball tasks with standard and deviant stimuli using both EEG and fNIRS data.

Main Results:

• The multimodal data fusion strategy improved classification accuracy by up to 8.40% (SVM) and 2.02% (GNN) compared to single modalities.
• The context-aware GNN outperformed baseline models, achieving 5.3% higher accuracy for EEG, 4.07% for fNIRS, and 4.53% for multimodal data.

Conclusions:

• Multimodal fusion of EEG and fNIRS signals significantly enhances BCI performance.
• The developed context-aware GNN model effectively leverages inter-subject relationships for improved neural signal classification.