A high-performance neuroprosthesis for speech decoding and avatar control

Sean L Metzger ^1,2,3, Kaylo T Littlejohn ^1,2,4, Alexander B Silva ^1,2,3

⁰Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.

Nature +

|

August 23, 2023

View abstract on PubMed

Summary

This study developed a brain-computer interface for restoring communication in paralysis. It decodes silent speech into text, audio, and avatar animation with high accuracy and speed.

Area Of Science

Neuroscience
Biomedical Engineering
Artificial Intelligence

Background

Restoring communication for individuals with paralysis is a significant challenge.
Existing speech neuroprostheses often lack naturalistic speed and expressivity.
Severe limb and vocal paralysis severely limits communication abilities.

Purpose Of The Study

To develop a high-performance, real-time speech neuroprosthesis.
To enable multimodal communication output (text, speech audio, facial animation).
To restore embodied communication for individuals with severe paralysis.

Main Methods

High-density surface electrocorticography (ECoG) recordings from the speech cortex.
Deep learning models trained on neural data from attempted silent speech.
Real-time decoding across text, synthesized speech, and avatar animation modalities.

Main Results

Achieved rapid, large-vocabulary text decoding at 78 words per minute with 25% word error rate.
Demonstrated intelligible and personalized speech synthesis, matching the participant's pre-injury voice.
Enabled control of facial-avatar animation for speech and non-speech gestures.
High decoder performance achieved within two weeks of training.

Conclusions

A multimodal speech-neuroprosthetic approach shows significant promise for restoring communication.
This technology can provide full, embodied communication for individuals with severe paralysis.
Rapid training and high performance indicate clinical viability for brain-computer interfaces.