通过提取的面部和语音特征的基于注意力的融合来检测多模式情绪
Dilnoza Mamieva1, Akmalbek Bobomirzaevich Abdusalomov1, Alpamis Kutlimuratov2
1Department of Computer Engineering, Gachon University, Seongnam-si 13120, Republic of Korea.
Sensors (Basel, Switzerland)
|July 8, 2023
概括
使用融合面部和语音特征的多式情绪识别提高了准确性. 基于注意力的方法优先考虑信息输入,以便更有弹性和更全面地检测情绪状态.
科学领域:
- 人工智能的人工智能
- 人与计算机的交互
- 情感计算是一种情感计算.
背景情况:
- 单模态情绪检测的准确性和弹性不如多模态方法.
- 情绪通过各种互补的模式传达,需要合并分析才能获得完整的画面.
- 现有的多式联运系统可以通过选择性地专注于最有信息的功能来改进.
研究的目的:
- 开发一种新的基于注意力的方法,用于多式联络式情绪识别.
- 整合和选择性地融合面部和语音特征,以提高准确性.
- 通过结合低级和高级特征来增强情绪状态的表现.
主要方法:
- 利用独立的编码器提取面部和语音特征.
- 实施了注意力机制,从每个模式中选择最有信息的功能.
- 采用融合网络来结合功能和用于情感识别的分类层.
- 整合了低级和高级的面部特征,用于全面的表情分析.
主要成果:
- 提出的基于注意力的系统在IEMOCAP和CMU-MOSEI数据集上取得了卓越的性能.
- 在IEMOCAP上获得了74.6%的加权精度 (WA) 和66.1%的F1得分.
- 在CMU-MOSEI上获得了80.7%的WA和73.7%的F1得分.
- 通过专注于最突出的语音和面部表情输入,表现出更好的准确性.
结论:
- 基于注意力的多式联动情绪识别系统显著优于现有模型.
- 选择性特征融合和全面的特征表示增强了情绪检测能力.
- 这种方法为理解情绪状态提供了更准确和更有弹性的方法.
相关概念视频
01:21Association Areas of the Cortex
5.5K
Association areas are regions of the cerebral cortex that do not have a specific sensory or motor function. Instead, they integrate and interpret information from various sources to enable higher cognitive processes such as memory, learning, and decision-making. Some key association areas include the following:
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
5.5K
01:24Facial Feedback Hypothesis
193
Charles Darwin proposed that facial expressions are an evolutionary adaptation for communication. He argued that these expressions are not influenced by culture but are universal across species. For example, a snarling expression with exposed teeth signals a threat in many animals, including humans. Darwin also suggested that displaying an emotion can intensify the feeling. Smiling, for example, could enhance one's sense of happiness. This idea laid the foundation for understanding the role...
193
01:20Labeling Emotion
185
Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
185
01:20Physiology of Emotion
996
The physiology of emotions is a multifaceted process involving the autonomic nervous system, brain structures, hormones, and neurotransmitters. This intricate interplay dictates how emotions manifest in the body and influence behavior.
Autonomic Nervous System
The autonomic nervous system (ANS) plays a critical role in emotional responses by regulating involuntary physiological functions. It consists of two main components: the sympathetic and parasympathetic systems. The sympathetic system...
Autonomic Nervous System
The autonomic nervous system (ANS) plays a critical role in emotional responses by regulating involuntary physiological functions. It consists of two main components: the sympathetic and parasympathetic systems. The sympathetic system...
996