Emotion-BIND for multimodal emotion recognition and reasoning

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal Emotion Recognition in Conversation (MERC) models are crucial for understanding human affect across various data types.
• Existing MERC models face challenges with feature degradation and modality misalignment due to linear alignment and frozen encoders.
• Conventional 1D positional encoding limits the modeling of complex visual and temporal dynamics.

Purpose of the Study:

• To introduce Emotion-BIND, a novel multimodal framework for enhanced emotion recognition in conversations.
• To address limitations in cross-modal feature alignment and temporal information modeling in existing MERC approaches.
• To leverage ImageBind representations and a differentiated positional encoding strategy (m-ROPE) for improved performance.

Main Methods:

• Developed Emotion-BIND, a framework integrating ImageBind-based representations.
• Implemented a differentiated positional encoding strategy, m-ROPE, with 2D RoPE for images and 3D RoPE for videos.
• Mapped features from text, audio, image, and video modalities into a shared embedding space for unified processing.

Main Results:

• Emotion-BIND achieved a Weighted Average Recall (WAR) of 71.00% and an F1 score of 90.05% on the DFEW dataset, outperforming compared methods in these metrics.
• On the MELD dataset, the framework obtained a WAR of 67.88% and an F1 score of 91.02%.
• While achieving high WAR and F1 scores, the Unweighted Average Recall (UAR) on DFEW was lower than the top-performing model.

Conclusions:

• Emotion-BIND demonstrates a promising approach for multimodal emotion recognition, particularly in improving cross-modal alignment and temporal modeling.
• The integration of ImageBind and m-ROPE offers significant advantages over conventional methods for affective computing.
• The framework's performance highlights the potential for more sophisticated multimodal fusion techniques in understanding conversational emotions.