Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language.

Area of Science:

• Neuroscience
• Speech Technology
• Brain-Computer Interfaces

Background:

• Silent speech decoding (SSD) utilizes articulatory neuromuscular activities for brain-computer interfaces (BCIs).
• Decoding surface electromyography (sEMG) for SSD is an active research area.
• Restoring silent speech in tonal languages like Mandarin Chinese presents significant challenges.

Purpose of the Study:

• To propose an optimized sequence-to-sequence (Seq2Seq) approach for synthesizing voice from sEMG-based silent speech.
• To address the difficulties in decoding silent speech for Mandarin Chinese.
• To improve the accuracy and naturalness of synthesized speech from articulatory signals.

Main Methods:

• Extraction of duration information from audio length to regulate sEMG-based silent speech.
• Implementation of a deep-learning encoder-decoder model.
• Utilization of a state-of-the-art vocoder for audio waveform generation.

Main Results:

• Successful decoding of silent speech in Mandarin Chinese across six speakers.
• Achievement of an average character error rate (CER) of 6.41%.
• Validation of the model's effectiveness through human evaluation.

Conclusions:

• The proposed optimized Seq2Seq model effectively synthesizes voice from sEMG signals for Mandarin Chinese silent speech.
• The method demonstrates significant potential for advancing BCIs and speech restoration technologies.
• This work contributes to overcoming challenges in decoding tonal languages for silent speech applications.