Hierarchical Attention-Based Multimodal Fusion Network for Video Emotion Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Analysis

Background:

• Contextual information like scenes and objects significantly impacts video emotion recognition accuracy.
• Existing methods often overlook the varying emotional cues present across different images and modalities.
• Improving multimodal fusion is crucial for accurate video-based emotion understanding.

Purpose of the Study:

• To propose a hierarchical attention-based multimodal fusion network for enhanced video emotion recognition.
• To address the challenge of differing emotional clues across various images and modalities.
• To improve the accuracy and robustness of emotion recognition in videos.

Main Methods:

• Developed a multimodal feature extraction module with subnetworks for facial, scene, and global image features.
• Implemented a hierarchical attention mechanism to aggregate features and emotion scores within and across modalities.
• Designed a multimodal feature fusion module to generate the final video emotion representation.

Main Results:

• The proposed hierarchical attention network effectively extracts and fuses multimodal features.
• Experimental results demonstrate significant improvements in video emotion recognition accuracy.
• The method successfully addresses the issue of varying emotional clues in different images and modalities.

Conclusions:

• The hierarchical attention-based multimodal fusion network offers a promising approach for video emotion recognition.
• Incorporating context and employing attention-based fusion enhances the understanding of video-based emotions.
• The proposed method provides a robust and effective solution for complex emotion recognition tasks in videos.