通过高效的声学-语言双通道框架,高保真度的神经语音重建
Jiawei Li1,2, Chunxu Guo1, Chao Zhang3,4
1School of Biomedical Engineering, ShanghaiTech University, Shanghai, China.
eLife
|March 5, 2026
概括
研究人员开发了一种双路径框架,从电皮质谱 (ECoG) 数据中重建语音. 这种方法成功地解码了声学和语言特征,克服了大脑计算机接口 (BCI) 语音合成的先前局限性.
科学领域:
- 神经科学是一个神经科学.
- 人工智能的人工智能
- 生物医学工程 生物医学工程
背景情况:
- 从神经信号进行语音重建对于理解语音处理和推进脑计算机接口 (BCI) 至关重要.
- 当前的方法往往会牺牲声学细节 (如音调和曲调),以获得语言清晰度 (词和音符),或者反之亦然.
- 这种权衡限制了合成语音的自然性和表现力.
研究的目的:
- 引入一种新的双路径框架,用于从神经记录中同时解码声学和语言信息.
- 克服现有的语音重建技术中固有的声学语言权衡.
- 从电皮质谱 (ECoG) 数据中合成自然听起来和可理解的语音.
主要方法:
- 设计了一个双路径框架,其中包括一个声学路径 (LSTM解码器和HiFi-GAN) 用于光谱时间特征,一个语言路径 (变压器适配器和TTS发生器) 用于词令牌.
- 这两种途径使用语音克隆进行了整合,以确保声学和语言的准确性.
- 该框架使用有限的电皮质谱 (ECoG) 数据 (每人20分钟) 进行训练和评估.
主要成果:
- 提出的方法实现了高度可理解的合成语音,平均意见分数为4.0/5.0.
- 记录了18.9%的低单词错误率,表明了显著的语言准确性.
- 该框架成功地重建了平衡自然声学品质和语言正确性的语音.
结论:
- 双路径框架有效地解决了神经语音解码中的声学-语言权衡问题.
- 这种方法可以从ECoG数据中重建自然和可理解的语音.
- 这些发现对推进BCI和理解神经语音编码具有重大意义.
相关概念视频
01:15Auditory Pathway
8.2K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
8.2K
01:10Reconstruction of Signal using Interpolation
804
Signal processing techniques are essential for accurately converting continuous signals to digital formats and vice versa. When a continuous signal is sampled with a period T, the resulting sampled signal exhibits replicas of the original spectrum in the frequency domain, spaced at intervals equal to the sampling frequency. To handle this sampled signal, a zero-order hold method can be applied, which creates a piecewise constant signal by retaining each sample's value until the next...
804
01:25Neural Circuits
3.1K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
3.1K