A neural speech decoding framework leveraging deep learning and speech synthesis.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Biomedical Engineering

Background:

• Restoring speech for individuals with neurological deficits via brain-computer interfaces (BCIs) is crucial but challenging.
• Difficulties include limited neural signal data, complex data, and high dimensionality.
• Existing methods face limitations in real-time application and broad applicability.

Purpose of the Study:

• To present a novel deep learning framework for decoding human speech from neural signals.
• To enable the translation of electrocorticographic (ECoG) signals into speech parameters and synthesize natural-sounding speech.
• To facilitate the development of real-time neural prostheses for speech restoration.

Main Methods:

• Developed a deep learning framework comprising an ECoG decoder and a differentiable speech synthesizer.
• Utilized a speech-to-speech auto-encoder with a speech encoder and the speech synthesizer to generate reference speech parameters.
• Trained models using ECoG signals and corresponding speech data from 48 participants.

Main Results:

• The framework successfully decodes speech from ECoG signals, generating natural-sounding and reproducible speech.
• High correlation in speech decoding was achieved, even with causal operations suitable for real-time prostheses.
• Speech decoding was successful in participants with both left and right hemisphere ECoG coverage.

Conclusions:

• The novel deep learning framework offers a promising approach for neural speech decoding.
• The system's ability to generate natural speech and its reproducibility across participants highlight its potential.
• This technology could significantly advance the development of speech prostheses for individuals with speech impairments due to neurological damage.