DER-GCN: Dialog and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialog Emotion...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Multimodal dialog emotion recognition (MDER) is crucial for understanding human interaction.
• Existing MDER methods often overlook the impact of event relations on emotional expression.
• Deep learning (DL) advancements provide opportunities to improve MDER.

Purpose of the Study:

• To propose a novel deep learning model for MDER that incorporates both dialog and event relations.
• To enhance the fusion of multimodal features by considering inter-speaker and event dependencies.
• To improve the representation learning of minority emotion classes in dialogs.

Main Methods:

• Developed a Dialog and Event Relation-aware Graph Convolutional Neural Network (DER-GCN).
• Constructed a weighted multirelationship graph to model speaker and event dependencies.
• Introduced a self-supervised masked graph autoencoder (SMGAE) for feature fusion.
• Designed a multiple information Transformer (MIT) to capture cross-relational correlations.
• Implemented a contrastive learning-based loss strategy for minority class enhancement.

Main Results:

• The DER-GCN model demonstrated significant improvements on benchmark datasets (IEMOCAP, MELD).
• Achieved higher average accuracy and overall emotion recognition performance compared to existing methods.
• Effectively captured latent event relations and improved multimodal feature fusion.

Conclusions:

• The proposed DER-GCN model advances the state-of-the-art in multimodal dialog emotion recognition.
• Integrating dialog and event relations is vital for accurate emotion understanding.
• The model's architecture effectively handles complex multimodal dependencies and improves robustness.