Related Experiment Video
Updated: May 25, 2025
05:36
Central and Divided Visual Field Presentation of Emotional Images to Measure Hemispheric Differences in Motivated Attention
Published on: November 16, 2017
7.5K
[Audiovisual emotion recognition based on a multi-head cross attention mechanism]
Ziqiong Wang1, Dechun Zhao1, Lu Qin1
1School of Life Health Information Science and Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China.
Summary
This study introduces a novel cross-modal audiovisual emotion recognition model using multi-head cross-attention. The model effectively integrates audio and video data, achieving high F1 scores and addressing modality missing challenges.
Area of Science:
- Computer Science
- Artificial Intelligence
- Machine Learning
Context:
- Representational learning is crucial for accurate audiovisual emotion recognition.
- Existing methods face challenges in creating affective representations with both consistency and variability.
- Effective fusion of audio and visual cues remains a significant hurdle.
Purpose:
- To propose a novel cross-modal audiovisual emotion recognition model.
- To achieve fused feature and modality alignment using a multi-head cross-attention mechanism.
- To address the challenge of missing modalities through a segmented training strategy.
Summary:
- A multi-head cross-attention model was developed for audiovisual emotion recognition.
- The model employs a segmented training strategy and a unimodal auxiliary loss task.
- It effectively captures intra- and inter-modal feature representations, preserving independent modality information.
Impact:
- Achieved macro and micro F1 scores of 84.5% and 88.2% on the CREMA-D dataset.
- Successfully unifies unimodal and multimodal emotion recognition frameworks.
- Offers a new solution for accurate audiovisual emotion recognition.
