Optimizing Deep Neural Networks for EEG-Based Speech Recognition: A Multimodal Approach to Assistive Communication

Area of Science:

• Neuroscience
• Artificial Intelligence
• Assistive Technology

Background:

• Speech recognition for individuals with impairments is challenging due to atypical speech patterns.
• Traditional acoustic-only models struggle with these variations.
• There is a need for robust and efficient speech recognition solutions for impaired communication.

Purpose of the Study:

• To introduce NeuroSpeech, a novel multimodal framework integrating electroencephalography (EEG) and acoustic features.
• To enhance speech recognition accuracy, robustness, and efficiency for individuals with speech impairments.
• To investigate the role of EEG as a noise-robust complementary signal.

Main Methods:

• Developed NeuroSpeech, a multimodal framework combining EEG and acoustic features.
• Utilized a large-scale random search to identify optimal EEG encoder configurations and feature extraction parameters.
• Employed Explainable AI (XAI) methods, specifically SHAP, for model interpretability.
• Evaluated performance on Spanish (UNLP-CONICET) and English (KaraOne) datasets under clean and noisy conditions.

Main Results:

• NeuroSpeech achieved near-perfect accuracy in clean conditions (F1=0.986 Spanish; 0.837 English).
• Maintained strong performance in noisy conditions (F1=0.92 Spanish; 0.70 English), outperforming traditional models like Whisper.
• Demonstrated EEG's effectiveness as a noise-robust complementary signal.
• Showcased NeuroSpeech's lightweight nature (1-30M parameters) and near-real-time inference (10-18ms/sample).

Conclusions:

• NeuroSpeech significantly improves speech recognition for individuals with impairments by integrating neural and acoustic data.
• The framework offers enhanced robustness in noisy environments, a critical factor for real-world applications.
• NeuroSpeech represents a promising advancement in assistive technologies, enabling better communication for those with speech disorders.