Cross-modal gated feature enhancement for multimodal emotion recognition in conversations

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Computational Linguistics

Background:

• Emotion recognition in conversations (ERC) is crucial for empathetic AI applications like customer service and health monitoring.
• Current multimodal ERC methods struggle with feature extraction, underutilizing complementary data, and inter-modal redundancy.
• Effective multimodal emotion recognition requires robust fusion of diverse sensory inputs.

Purpose of the Study:

• To develop an advanced multimodal emotion recognition system using a novel cross-modal gated attention mechanism.
• To enhance the accuracy and stability of emotion recognition by effectively fusing visual, auditory, and textual features.
• To address challenges in feature extraction, complementary feature utilization, and inter-modal redundancy in current ERC research.

Main Methods:

• Extraction and fusion of visual, textual, and auditory features using a cross-modal gated attention mechanism.
• Implementation of a cross-modal guided gating mechanism to strengthen single-modality features and improve bimodal fusion.
• Proposal of a cross-modal distillation loss function with dual supervision to reduce redundancy and enhance feature discrimination.

Main Results:

• The proposed method achieved higher accuracy compared to existing approaches on the IEMOCAP and MELD datasets.
• The system demonstrated comparable F1 scores, indicating effective multimodal dependency capture.
• Experimental results validate the effectiveness of the gated attention and distillation loss in balancing modality contributions.

Conclusions:

• The developed cross-modal gated attention mechanism significantly improves multimodal emotion recognition accuracy and stability.
• The novel approach effectively addresses key challenges in feature fusion and redundancy reduction.
• This research contributes to the advancement of empathetic AI systems through enhanced conversational emotion understanding.