Adaptive Graph Learning with Multimodal Fusion for Emotion Recognition in Conversation

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Natural Language Processing

Background:

• Conversational emotion recognition is crucial for natural human-computer interaction.
• Existing methods struggle with the dual influence of global topic flow and local speaker interactions.
• Robust emotion recognition requires understanding complex conversational dependencies.

Purpose of the Study:

• To introduce GASMER (Graph-Adaptive Structure for Multimodal Emotion Recognition), a unified architecture for conversational emotion recognition.
• To address the challenges posed by global topic flow and local speaker-to-speaker dependencies.
• To enhance the accuracy and robustness of multimodal emotion recognition in conversations.

Main Methods:

• Developed GASMER, a novel architecture utilizing graph neural networks (GNNs) to model conversational dependencies.
• Implemented an adaptive graph learning mechanism within the GNN framework.
• Employed fine-grained multimodal fusion techniques.

Main Results:

• GASMER outperforms existing graph-based approaches in conversational emotion recognition.
• The model achieves competitive performance compared to recent multimodal fusion models.
• On the IEMOCAP dataset, GASMER improved accuracy by 2.7% and weighted F1-score by 3.6%.
• On the MOSEI dataset, GASMER achieved a 1.2% gain in binary classification accuracy (ACC-2).

Conclusions:

• Combining fine-grained multimodal fusion with adaptive graph learning is vital for effective conversational emotion recognition.
• GASMER demonstrates the efficacy of adaptive graph learning for modeling complex conversational dynamics.
• The proposed architecture offers a significant advancement in the field of emotion recognition.