在杂的环境中使用基于情感的上下文线索进行音视觉语音增强
Tassadaq Hussain1, Nasir Saleem1, Kia Dashtipour1
1School of Computing Engineering and the Built Environment, Edinburgh Napier University, Edinburgh, EH105DT, United Kingdom.
The Journal of the Acoustical Society of America
|January 16, 2026
概括
这项研究介绍了一种情感感知音视听语音增强 (EAVSE) 系统. 从面部标志中整合情绪线索显著提高了在噪音条件下语音清晰度.
科学领域:
- 信号处理 信号处理
- 人工智能的人工智能
- 人与计算机的交互
背景情况:
- 背景噪音在现实世界中严重影响了语音可理解性.
- 目前的视听语音增强 (AVSE) 方法与动态,杂的环境作斗争.
- 情感上下文是语言增强中未被充分利用的特征.
研究的目的:
- 提出和评估一种新的情感感知音视听语音增强 (EAVSE) 系统.
- 调查将面部标志的情绪特征纳入AVSE的影响.
- 在具有挑战性的声学条件下提高语音清晰度和可理解性.
主要方法:
- 开发了一个基于深度学习的EAVSE系统,利用听觉,视觉和情感信息.
- 从面部地标中提取情感特征,并将它们与视听数据融合在一起.
- 采用基于UNet的编码器-解码器网络进行联合多模式学习.
- 为了优化,利用了受扭曲影响的损失函数,特别是尺度不变信号与扭曲比率 (S-SNR).
- 在卡内基梅隆大学多式联络意见情感和情感强度数据集上训练和评估模型.
主要成果:
- 在客观指标 (PESQ,STOI) 和主观语音质量方面,EAVSE系统取得了显著的改进.
- 与基准AVSE和仅音频系统相比,表现出优异的性能,特别是在低SNR环境 (≤ -7.5dB) 中.
- 达到了7.32%的D STOI,0.33的D PESQ和7.8dB的D S-SNR,比0dB的SNR.
结论:
- 从面部标志中获得的情感上下文线索对于改善视听语言增强是有效的.
- 拟议的EAVSE系统为噪音和动态环境中的语音增强提供了强大的解决方案.
- 这些发现突显了多模式深度学习的潜力,将情感理解整合为更好的人机交互.
更多相关视频
05:51Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury
Published on: May 15, 2016
9.4K
11:39Assessment of Audio-Tactile Sensory Substitution Training in Participants with Profound Deafness Using the Event-Related Potential Technique
Published on: September 7, 2022
2.6K
相关概念视频
Non-Verbal Cues
286
Non-verbal communication extends beyond gestures and facial expressions to include vocal elements known as paralanguage. Paralanguage consists of non-verbal vocal cues such as pitch, loudness, speech rate, pauses, and non-verbal vocalizations like laughter, sighs, and moans. These elements not only accompany speech but also provide critical emotional and contextual information.The Role of Paralanguage in CommunicationParalanguage adds depth to spoken language by conveying emotions and...
286
Perceiving Loudness, Pitch, and Location
942
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
942
Labeling Emotion
604
Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
604
Emotional Expression
956
Emotional expression encompasses how individuals convey their emotions through verbal communication and non-verbal cues. These non-verbal actions include facial expressions, body language, and physical gestures, such as frowning or smiling. Among these, facial expressions play a crucial role in emotional expression and are understood universally, indicating a biological basis for how humans communicate emotions.
Universal Facial Expressions
Psychologist Paul Ekman identified seven basic...
Universal Facial Expressions
Psychologist Paul Ekman identified seven basic...
956
Coping Strategies: Emotion Focused
443
Emotion-focused coping refers to a set of strategies aimed at managing the emotional impact of stressors, rather than directly addressing their causes. This approach involves altering one's emotional response to stressful situations to reduce their psychological effects. For example, individuals might talk with a friend or engage in activities like journaling to express their feelings. Such actions can help achieve emotional clarity or release, providing the psychological stability needed...
443
Auditory Perception
1.0K
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
1.0K
