Emotion-BIND for multimodal emotion recognition and reasoning

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal Emotion Recognition in Conversation (MERC) models are crucial for understanding human affective states.
• Existing MERC models face challenges with feature degradation and modality misalignment due to linear alignment and frozen encoders.
• Conventional 1D positional encoding limits effective modeling of visual and temporal dynamics.

Purpose of the Study:

• To introduce Emotion-BIND, a novel multimodal framework designed to overcome limitations in current MERC models.
• To enhance cross-modal feature alignment and temporal modeling for improved emotion recognition accuracy.
• To leverage ImageBind representations and a differentiated positional encoding strategy (m-ROPE) for superior performance.

Main Methods:

• Emotion-BIND integrates ImageBind-based representations into a unified framework.
• A differentiated positional encoding strategy, m-ROPE, is employed, utilizing 2D RoPE for images and 3D RoPE for videos.
• Features from text, audio, image, and video modalities are mapped into a shared embedding space for comprehensive analysis.

Main Results:

• On the DFEW dataset, Emotion-BIND achieved a Weighted Average Recall (WAR) of 71.00% and an F1 score of 90.05%, outperforming compared methods in these metrics.
• On the MELD dataset, the framework obtained a WAR of 67.88% and an F1 score of 91.02%.
• While achieving state-of-the-art WAR and F1 scores, the Unweighted Average Recall (UAR) on DFEW was lower than the best competitor.

Conclusions:

• Emotion-BIND demonstrates significant advancements in multimodal emotion recognition by effectively addressing cross-modal alignment and temporal modeling.
• The proposed m-ROPE strategy and ImageBind integration offer a robust approach for capturing complex affective cues across modalities.
• The framework's strong performance on benchmark datasets highlights its potential for real-world applications in understanding conversational emotions.