Hybrid EEG-fNIRS phoneme classification based on imagined and perceived speech

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Severe motor impairments limit communication, driving research into brain-computer interfaces (BCIs) for speech prosthesis.
• Existing BCIs often rely on unimodal neuroimaging (EEG or fNIRS), with multimodal approaches being less explored.

Purpose of the Study:

• To investigate the efficacy of a hybrid electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) system for phoneme decoding in imagined and perceived speech.
• To compare the performance of the hybrid system against unimodal EEG and fNIRS systems.

Main Methods:

• Offline phoneme decoding was performed using combined EEG and fNIRS data from 22 participants imagining and perceiving four phonemes (/a/, /i/, /b/, /k/).
• Features extracted included power spectral densities (EEG) and mean hemoglobin concentration changes (fNIRS).
• Mutual information criterion and 10-fold cross-validation were used for feature selection and model optimization.

Main Results:

• Hybrid classification achieved 77.29% accuracy for imagined speech and 76.05% for perceived speech.
• No significant difference in performance was observed between hybrid and EEG-only classification.
• fNIRS-based phoneme discrimination yielded lower accuracy compared to the hybrid and EEG-based methods.

Conclusions:

• This study presents an innovative application of multimodal EEG-fNIRS data for phoneme decoding in both imagined and perceived speech.
• The four-class imagined speech classification was predominantly driven by EEG features, outperforming previous studies.
• The findings suggest that hybrid EEG-fNIRS systems hold potential for developing intuitive speech prosthetics.