将听觉感知和自然语言处理与语义信息深度神经网络相结合
Michele Esposito1, Giancarlo Valente2, Yenisel Plasencia-Calaña3
1Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands. m.esposito@maastrichtuniversity.nl.
Scientific reports
|September 9, 2024
概括
这项研究表明,深度神经网络 (DNN) 可以通过使用语义信息更好地识别声音,类似于人类的做法. 这种方法通过捕捉声音之间的关系来改善人工听觉系统.
科学领域:
- 人工智能的人工智能
- 认知神经科学 认知神经科学
- 机器学习 机器学习
背景情况:
- 人类的声音识别轻松地利用语义信息,这是当前人工听觉系统缺乏的能力.
- 深度神经网络 (DNN),特别是CNN,在声音分类方面表现出色,但通常使用分类标签,忽视语义关系.
- 认知神经科学表明,人类利用语义信息与声学线索一起进行卓越的声音识别.
研究的目的:
- 调查将语义信息纳入DNN是否可以提高声音识别性能.
- 开发一种新的声音识别方法,更好地模拟人类的听觉感知.
- 为了弥合人工声音识别系统和人类听觉能力之间的差距.
主要方法:
- 以声音识别为回归问题,训练CNN将光谱图映射到连续的语义表示.
- 使用自然语言处理 (NLP) 模型,如Word2Vec,BERT和CLAP文本编码器用于语义嵌入.
- 在388,211个声音的大数据集上训练了两个DNN类型:semDNN (语义连续嵌入) 和catDNN (分类标签).
主要成果:
- 在四个外部数据集中,SemDNN表现出了比catDNN更好的性能,保留了更高层次的语义关系.
- 与catDNN和其他模型相比,SemDNN在自然声音的相似性评级中更好地近似了人类听众行为.
- 该研究证实了语义信息集成显著改善DNN声音识别的假设.
结论:
- 将语义信息纳入DNN对于推进人工听觉系统至关重要.
- 半DNN方法提供了更类似于人类的声音识别能力,超越了简单的分类分类.
- 这项研究强调了语义理解在开发更复杂的AI用于听觉感知方面的重要性.
更多相关视频
03:31Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications
Published on: December 15, 2023
487
11:39Assessment of Audio-Tactile Sensory Substitution Training in Participants with Profound Deafness Using the Event-Related Potential Technique
Published on: September 7, 2022
2.1K
相关概念视频
Perceiving Loudness, Pitch, and Location
202
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
202
Hearing
51.9K
When we hear a sound, our nervous system is detecting sound waves—pressure waves of mechanical energy traveling through a medium. The frequency of the wave is perceived as pitch, while the amplitude is perceived as loudness.
51.9K
Auditory Pathway
5.3K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
5.3K
Auditory Perception
324
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
324
Parallel Processing
146
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
146
Perception
442
Perception is a fundamental psychological process that enables individuals to organize, interpret, and consciously experience sensory information. This process is crucial for understanding and interacting with the world around us. It includes both bottom-up and top-down processing, each playing a distinct role in how we perceive our environment.
Bottom-up processing begins at the sensory level, where receptors detect external environmental stimuli. These could include the tactile sensation of...
Bottom-up processing begins at the sensory level, where receptors detect external environmental stimuli. These could include the tactile sensation of...
442
