Speech synthesis from neural decoding of spoken sentences

Area of Science:

• Neuroscience
• Biomedical Engineering
• Speech Science

Background:

• Neurological impairments can severely impact communication abilities.
• Restoring speech through technology is a significant challenge due to complex vocal tract control.

Purpose of the Study:

• To develop a neural decoder capable of synthesizing audible speech from brain activity.
• To leverage kinematic and sound representations in cortical activity for speech decoding.

Main Methods:

• Designed a recurrent neural network-based decoder to interpret neural signals.
• Decoded cortical activity into articulatory movement representations.
• Transformed articulatory representations into speech acoustics.

Main Results:

• Synthesized audible speech from neural activity was successfully identified and transcribed in closed-vocabulary tests.
• Incorporating intermediate articulatory dynamics improved performance, especially with limited data.
• Decoded articulatory representations demonstrated cross-speaker conservation, allowing decoder transferability.

Conclusions:

• The developed neural decoder effectively synthesizes speech from brain activity.
• This technology shows potential for a speech neuroprosthetic to restore communication.
• The decoder's ability to work with silent miming suggests versatile applications.