Optimizing Deep Neural Networks for EEG-Based Speech Recognition: A Multimodal Approach to Assistive Communication

Anarghya Das , Puru Soni , Hubin Zhao

IEEE Journal of Biomedical and Health Informatics +

|

December 8, 2025

View abstract on PubMed

Summary

This study introduces NeuroSpeech, a multimodal system combining electroencephalography (EEG) and acoustics for improved speech recognition in individuals with impairments. NeuroSpeech demonstrates robust performance, even in noisy conditions, offering a promising assistive technology.

Area Of Science

Neuroscience
Artificial Intelligence
Assistive Technology

Background

Speech recognition for individuals with impairments is challenging due to atypical speech patterns.
Traditional acoustic-only models struggle with these variations.
There is a need for robust and efficient speech recognition solutions for impaired communication.

Purpose Of The Study

To introduce NeuroSpeech, a novel multimodal framework integrating electroencephalography (EEG) and acoustic features.
To enhance speech recognition accuracy, robustness, and efficiency for individuals with speech impairments.
To investigate the role of EEG as a noise-robust complementary signal.

Main Methods

Developed NeuroSpeech, a multimodal framework combining EEG and acoustic features.
Utilized a large-scale random search to identify optimal EEG encoder configurations and feature extraction parameters.
Employed Explainable AI (XAI) methods, specifically SHAP, for model interpretability.
Evaluated performance on Spanish (UNLP-CONICET) and English (KaraOne) datasets under clean and noisy conditions.

Main Results

NeuroSpeech achieved near-perfect accuracy in clean conditions (F1=0.986 Spanish; 0.837 English).
Maintained strong performance in noisy conditions (F1=0.92 Spanish; 0.70 English), outperforming traditional models like Whisper.
Demonstrated EEG's effectiveness as a noise-robust complementary signal.
Showcased NeuroSpeech's lightweight nature (1-30M parameters) and near-real-time inference (10-18ms/sample).

Conclusions

NeuroSpeech significantly improves speech recognition for individuals with impairments by integrating neural and acoustic data.
The framework offers enhanced robustness in noisy environments, a critical factor for real-world applications.
NeuroSpeech represents a promising advancement in assistive technologies, enabling better communication for those with speech disorders.