High-fidelity neural speech reconstruction through an efficient acoustic-linguistic dual-pathway framework.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Biomedical Engineering

Background:

• Speech reconstruction from neural signals is vital for understanding speech processing and advancing brain-computer interfaces (BCIs).
• Current methods often sacrifice acoustic details (like pitch and prosody) for linguistic clarity (words and phonemes), or vice versa.
• This trade-off limits the naturalness and expressiveness of synthesized speech.

Purpose of the Study:

• To introduce a novel dual-path framework for concurrently decoding acoustic and linguistic information from neural recordings.
• To overcome the acoustic-linguistic trade-off inherent in existing speech reconstruction techniques.
• To synthesize natural-sounding and intelligible speech from electrocorticography (ECoG) data.

Main Methods:

• A dual-path framework was designed, featuring an acoustic pathway (LSTM decoder and HiFi-GAN) for spectrotemporal features and a linguistic pathway (transformer adaptor and TTS generator) for word tokens.
• The two pathways were integrated using voice cloning to ensure both acoustic and linguistic accuracy.
• The framework was trained and evaluated using limited electrocorticography (ECoG) data (20 minutes per subject).

Main Results:

• The proposed method achieved highly intelligible synthesized speech, with a Mean Opinion Score of 4.0/5.0.
• A low word error rate of 18.9% was recorded, indicating significant linguistic accuracy.
• The framework successfully reconstructed speech that balanced both natural acoustic qualities and linguistic correctness.

Conclusions:

• The dual-path framework effectively resolves the acoustic-linguistic trade-off in neural speech decoding.
• This approach enables the reconstruction of natural and intelligible speech from ECoG data.
• The findings hold significant implications for advancing BCIs and understanding neural speech coding.