从大脑活动中解码和合成音调语言语音
Yan Liu1,2,3,4, Zehao Zhao1,2,3,4, Minpeng Xu5,6
1Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, Shanghai 200040, China.
Science advances
|June 9, 2023
概括
这项研究介绍了一种新的脑计算机接口 (BCI),用于恢复语音语言说话者的语音. 新模型有效地解码和合成语音,包括词汇音调,为通信恢复提供希望.
科学领域:
- 神经科学是一个神经科学.
- 计算语言学 计算语言学
- 生物医学工程 生物医学工程
背景情况:
- 语音脑计算机接口 (BCI) 在非音调语言障碍中显示出恢复沟通的前景.
- 语音语言的语音BCI是复杂的,因为需要通过喉运动精确的词汇音调控制.
- 现有的模型往往忽略了音调语言生产的特定神经相关性.
研究的目的:
- 开发一个BCI模型,能够合成语音语言的语音.
- 为了应对解码词典音调与基本音节一起用于语音恢复的挑战.
- 调查神经科学启发的神经网络架构对音调语音BCI的有效性.
主要方法:
- 设计了一个模块化,多流神经网络,用于从内录音中直接进行音调语言语音合成.
- 使用并行神经网络流独立解码词汇音调和基本音节.
- 通过将解码的音色音节标签与非歧视性语音神经活动集成来合成语音.
主要成果:
- 与基线模型相比,拟议的模型实现了更高的性能.
- 演示了有效的语音语言语音解码和合成.
- 该模型需要适度的训练数据和计算资源.
结论:
- 开发的BCI模型为音色语言语音恢复提供了一个潜在的策略.
- 这种方法强调了针对与音调相关的皮质特征的重要性.
- 这些发现促进了临床上有效的语音BCI对不同语言群体的可行性.
相关概念视频
01:10Higher Mental Functions of the Brain: Language
939
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
939
01:31Hearing
52.5K
When we hear a sound, our nervous system is detecting sound waves—pressure waves of mechanical energy traveling through a medium. The frequency of the wave is perceived as pitch, while the amplitude is perceived as loudness.
52.5K
01:21Perceiving Loudness, Pitch, and Location
277
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
277
01:27Language and Cognition
381
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
381
01:19Encoding
210
Information enters the brain through encoding, which is the input of information into the memory system. Once sensory information is received from the environment, the brain labels or codes it. The information is then organized with similar information and connected to existing concepts. Encoding occurs through automatic processing and effortful processing.
Automatic processing involves the encoding of details like time, space, frequency, and the meaning of words, usually done without conscious...
Automatic processing involves the encoding of details like time, space, frequency, and the meaning of words, usually done without conscious...
210
01:22Neurons as Communicators of the Brain
1.3K
Neurons, the fundamental units of the brain and nervous system, function as the primary transmitters of information throughout the body. Their ability to communicate through electrical and chemical signals is vital for every bodily function, from regulating the heartbeat to processing complex thoughts. Each neuron has three main components: the cell body (soma), dendrites, and an axon, each specialized to facilitate swift and efficient neural communication.
Cell Body
The cell body, also known...
Cell Body
The cell body, also known...
1.3K