Improving inner speech decoding by hybridisation of bimodal EEG and fMRI data

Area of Science:

• Neuroscience
• Cognitive Science
• Machine Learning

Background:

• Inner speech, the internal monologue, is a key cognitive process.
• Decoding brain activity related to inner speech is challenging with unimodal neuroimaging.
• Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) offer complementary temporal and spatial information.

Purpose of the Study:

• To investigate the performance benefits of hybridizing fMRI and EEG data for inner speech decoding.
• To compare different data fusion strategies for multimodal brain data.
• To assess the efficacy of bimodal fusion over unimodal models in classifying inner speech.

Main Methods:

• Recorded simultaneous fMRI and EEG data from four participants during an inner speech task.
• Implemented two fusion approaches: probability vector concatenation and feature engineering-based data fusion.
• Developed and compared subject-dependent machine learning models for decoding an 8-word classification task.

Main Results:

• Hybrid fMRI-EEG fusion strategies significantly improved inner speech decoding performance across all participants.
• An average accuracy increase of 6.025% was observed for the bimodal fusion models compared to unimodal approaches.
• Participant-specific data structures influenced decoding performance in subject-dependent fusion models.

Conclusions:

• Hybridization of fMRI and EEG data provides superior performance for inner speech decoding compared to unimodal methods.
• The developed fusion strategies offer a promising avenue for advancing brain-computer interfaces and cognitive neuroscience research.
• Multimodal neuroimaging fusion is crucial for capturing the complex neural dynamics underlying inner speech.