Improving inner speech decoding by hybridisation of bimodal EEG and fMRI data

Decoding inner speech from the brain via the hybridisation of fMRI and EEG data is explored to investigate the performance benefits over unimodal models. Two different fusion approaches are examined: concatenation of probability vectors from unimodal fMRI and EEG machine learning models, and data fusion with feature engineering. Same-task inner speech data are recorded from four participants, and different processing strategies are compared and contrasted to previously-employed hybridisation efforts. Data across participants are discovered to encode different underlying structures, which correlates to decoding performances between subject-dependent fusion models. For all participants, the performance of inner speech decoding models is shown to improve when pursuing bimodal fMRI-EEG fusion strategies, with an average increase of 6.025% accuracy on an 8-word classification task across two semantic categories.