使用网络科学,用多层图形来检查视听语音感知.
Michael S Vitevitch1, Lorin Lachs2
1University of Kansas, Lawrence, KS, United States of America.
PloS one
|March 29, 2024
概括
本研究介绍了AV-net,这是视觉语音感知 (唇读) 的模型,展示了听觉和视觉信息如何相互作用以改善词识别. 该模型有助于理解唇读和开发辅助技术.
科学领域:
- 认知科学 认知科学
- 计算神经科学是一种神经科学.
- 语音处理 语音处理
背景情况:
- 视觉语音感知或唇读对于理解口语至关重要,特别是在杂的环境中.
- 现有的模型往往缺乏语音和视觉语音信息的整合.
- 了解听觉和视觉线索之间的相互作用是改善语音感知的关键.
研究的目的:
- 开发和评估模拟视觉语音感知的多层网络 (AV-net).
- 在各种条件下,将AV-net的性能与人类的唇读能力进行比较.
- 调查语音信息对视觉语音感知的影响.
主要方法:
- 创建了一个多层的AV网络,其中听觉和视觉层分别代表声学和视觉信息.
- 进行了激活扩散的计算机模拟,用于单词识别.
- 模拟结果与人类在视听,仅视听和仅视觉条件下的表现进行了比较.
- 对人类仅视觉读唇数据进行错误分析.
主要成果:
- 视听网络展示了听觉和视觉信息的整合,以增强单词识别.
- 模拟结果显示,在不同的呈现条件下,与人类参与者相比,性能可比.
- 语音信息在多层网络中显著影响了视觉语音感知.
结论:
- 该AV-net模型提供了关于视觉语音感知机制的见解.
- 这些发现对唇读训练和辅助技术的发展有影响.
- 该研究强调了整合语音和视觉信息对于强大的语音识别的重要性.
更多相关视频
09:27Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
Published on: October 13, 2018
10.0K
08:45Mapping Cortical Dynamics Using Simultaneous MEG/EEG and Anatomically-constrained Minimum-norm Estimates: an Auditory Attention Example
Published on: October 24, 2012
14.7K
相关概念视频
Perception of Sound Waves
4.5K
The human ear is not equally sensitive to all frequencies in the audible range. It may perceive sound waves with the same pressure but different frequencies as having different loudness. Moreover, the perception of sound waves depends on the health of an individual's ears, which decays with age. The health of one's ears may also be affected by regular exposure to loud noises.
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
4.5K
Signal Flow Graphs
217
Signal-flow graphs offer a streamlined and intuitive approach to representing control systems, providing an alternative to traditional block diagrams. These graphs use branches to symbolize systems and nodes to represent signals, effectively illustrating the relationships and interactions within the system.
In a signal-flow graph, branches denote the system's transfer functions, while nodes represent the signals. The direction of signal flow is indicated by arrows, with the corresponding...
In a signal-flow graph, branches denote the system's transfer functions, while nodes represent the signals. The direction of signal flow is indicated by arrows, with the corresponding...
217
Perceiving Loudness, Pitch, and Location
211
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
211
Parallel Processing
150
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
150
