一个深度神经网络训练一致的视听语言报告麦格格尔克效应
Haotian Ma1, Zhengjia Wang1, Xiang Zhang1
1Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA.
bioRxiv : the preprint server for biology
|September 2, 2025
概括
像AVHuBERT这样的深度神经网络可以复制麦格鲁克效应,从不一致的视听语言中感知虚幻音节. 这表明DNN可能有助于理解人类视听语音感知机制.
科学领域:
- 认知科学
- 神经科学
- 人工智能
背景情况:
- 麦克格尔克效应表明虚幻的音节感知来自不匹配的听觉和视觉语言输入.
- 目前的理论建议在不兼容的视听表现之间达成共识.
- 深度神经网络 (DNN) 提供了一种新的语言感知模式.
研究的目的:
- 调查一个深度神经网络,AVHuBERT,训练一致的视听语音,可以展示麦格鲁克效应.
- 为了比较AVHuBERT的表现与人类的感知.
- 探索DNN在理解视听语音处理方面的潜力.
主要方法:
- AVHuBERT,一个在音频视觉语音上受过训练的DNN,用一致和不一致的麦格格尔克刺激进行了测试.
- 基于仅听力,仅视觉和视听语音的转录准确性进行评价.
- 创建了100个AVHuBERT变体以评估麦格尔克效应的可变性.
主要成果:
- AVHuBERT 实现了对音频视觉语音的高准确性,反映了人类的表现模式 (音频 > 听觉 > 视觉).
- 当与麦格格尔克刺激 (听觉"ba",视觉"ga") 呈现时,AVHuBERT以25%的速度产生了融合感知"da".
- AVHuBERT变体对一致的刺激呈现一致的准确性,但对麦格鲁克的刺激具有很高的可变性,类似于人类.
结论:
- AVHuBERT能够复制麦格鲁克效应, 这表明DNN可以模拟复杂的视听语言现象.
- 这项研究强调了使用DNN作为研究人类语音感知神经和感知基础的工具的潜力.
- 这些发现支持人工智能模型在提高我们对人类感官处理和认知效应的理解方面的实用性.
相关概念视频
Facial Feedback Hypothesis
244
Charles Darwin proposed that facial expressions are an evolutionary adaptation for communication. He argued that these expressions are not influenced by culture but are universal across species. For example, a snarling expression with exposed teeth signals a threat in many animals, including humans. Darwin also suggested that displaying an emotion can intensify the feeling. Smiling, for example, could enhance one's sense of happiness. This idea laid the foundation for understanding the role...
244
Auditory Perception
580
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
580
Perception of Sound Waves
4.6K
The human ear is not equally sensitive to all frequencies in the audible range. It may perceive sound waves with the same pressure but different frequencies as having different loudness. Moreover, the perception of sound waves depends on the health of an individual's ears, which decays with age. The health of one's ears may also be affected by regular exposure to loud noises.
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
The pitch of a sound depends on the frequency and the pressure amplitude of the source. Two sounds of the same...
4.6K
Perceiving Loudness, Pitch, and Location
420
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
420
Auditory Pathway
5.8K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
5.8K
Nonconscious Mimicry
4.6K
Nonconscious mimicry occurs when individuals alter their mannerisms to match the behaviors and expressions of those nearby, without intention.
4.6K


