Speech synthesis from ECoG using densely connected 3D convolutional neural networks.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Speech Technology

Background:

• Direct speech synthesis from neural signals offers communication solutions for individuals with neurological impairments.
• Electrocorticography (ECoG) provides high-resolution neural data suitable for decoding complex speech production.
• Existing speech decoding models often struggle with the intricate dynamics of neural activity and continuous speech.

Purpose of the Study:

• To investigate the efficacy of deep neural networks in mapping ECoG signals to speech representations.
• To develop a novel method for reconstructing audible speech from neural recordings during speech production.

Main Methods:

• Utilized a densely connected convolutional neural network architecture to process ECoG data.
• Mapped neural signals to logMel spectrograms, an intermediate speech representation.
• Employed a Wavenet vocoder, conditioned on logMel features, to synthesize audible speech waveforms.

Main Results:

• Achieved high correlations (up to r=0.69) between reconstructed and original logMel spectrograms in six participants.
• Successfully synthesized audible speech with natural acoustic qualities using a Wavenet vocoder.
• Demonstrated the capability of deep learning to reconstruct speech from neural recordings.

Conclusions:

• Deep neural networks can effectively decode speech from ECoG signals during speech production.
• This approach represents a significant advancement in brain-computer interfaces for communication.
• The study highlights the potential for high-quality speech reconstruction from neural data.