Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Affective Computing

Background:

• Automated emotion recognition is crucial for emerging technologies.
• Existing methods often rely on single data modalities, limiting performance.
• Conversational emotion recognition presents unique challenges.

Purpose of the Study:

• To develop a robust multimodal approach for emotion recognition during conversations.
• To investigate the effectiveness of fusing audio, video, and text data.
• To compare multimodal performance against unimodal models and state-of-the-art.

Main Methods:

• Developed separate models for audio, video, and text modalities.
• Utilized a transformer-based cross-modality fusion with the EmbraceNet architecture.
• Fine-tuned models on the Multimodal Emotion Lines Dataset (MELD).

Main Results:

• The proposed multimodal network achieved up to 65% accuracy.
• Multimodal fusion significantly surpassed the performance of individual unimodal models.
• The model demonstrated robustness and outperformed state-of-the-art approaches on the MELD dataset.

Conclusions:

• Multimodal fusion is a superior strategy for conversational emotion recognition.
• The EmbraceNet architecture effectively integrates diverse data streams for emotion estimation.
• This research advances the field of automated emotion recognition with practical applications.