Decoding and synthesizing tonal language speech from brain activity.

Area of Science:

• Neuroscience
• Computational Linguistics
• Biomedical Engineering

Background:

• Speech brain-computer interfaces (BCIs) show promise for communication restoration in non-tonal language disorders.
• Tonal language speech BCI is complex due to the need for precise lexical tone control via laryngeal movements.
• Existing models often overlook the specific neural correlates of tonal language production.

Purpose of the Study:

• To develop a BCI model capable of synthesizing speech in tonal languages.
• To address the challenge of decoding lexical tones alongside base syllables for speech restoration.
• To investigate the efficacy of a neuroscience-inspired neural network architecture for tonal speech BCI.

Main Methods:

• Designed a modularized, multistream neural network for direct tonal language speech synthesis from intracranial recordings.
• Independently decoded lexical tones and base syllables using parallel neural network streams.
• Synthesized speech by integrating decoded tonal syllable labels with non-discriminant speech neural activity.

Main Results:

• The proposed model achieved higher performance compared to baseline models.
• Effective decoding and synthesis of tonal language speech were demonstrated.
• The model required modest training data and computational resources.

Conclusions:

• The developed BCI model offers a potential strategy for tonal language speech restoration.
• This approach highlights the importance of targeting tonal-related cortical features.
• The findings advance the feasibility of clinically valid speech BCIs for diverse linguistic populations.