Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition

Area of Science:

• Multimodal emotion recognition
• Artificial intelligence
• Human-computer interaction

Background:

• Emotion recognition in dyadic communication faces challenges due to heterogeneous data formats, feature fusion complexities, and annotation disagreements.
• Previous methods struggle with efficient fusion and accurate emotion prediction in real-world scenarios.

Purpose of the Study:

• To propose an efficient dyadic fusion network for improved multimodal emotion recognition.
• To address challenges in feature extraction, fusion, and annotation disagreement in dyadic communication.

Main Methods:

• Developed an efficient dyadic fusion network utilizing multiple sub-view attention layers for sequential utterance dependencies.
• Introduced a learnable mutual correlation factor for enhanced cross-modal feature fusion.
• Transformed categorical emotion prediction into a regression problem by embedding annotator labels into a k-dimensional vector to handle label disagreement.

Main Results:

• The proposed model significantly outperforms previous state-of-the-art methods on IEMOCAP and MELD datasets, achieving accuracy improvements of 3.8%-7.5%.
• The attention-based approach enhances model efficiency compared to recurrent neural network methods.
• The regression-based label handling improves annotation accuracy and data utilization.

Conclusions:

• The efficient dyadic fusion network offers a superior and more efficient approach to multimodal emotion recognition.
• The novel methods for feature fusion and handling annotation disagreement contribute to more robust emotion prediction models.
• This research advances the field of emotion recognition in dyadic communication, paving the way for more accurate human-computer interaction.