Sparse Spatial-Temporal Emotion Graph Convolutional Network for Video Emotion Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Video emotion recognition is gaining traction, but current methods often overlook crucial temporal context and inter-frame relationships.
• Existing spatial feature-based approaches face limitations due to the lack of comprehensive contextual understanding in videos.

Purpose of the Study:

• To propose a novel Sparse Spatial-Temporal Emotion Graph Convolutional Network (SE-GCN) for improved video emotion recognition.
• To address the limitations of existing methods by incorporating spatial-temporal relationships and context.

Main Methods:

• Constructing a sparse spatial graph based on emotional relationships between region proposals.
• Building a sparse temporal graph using emotion proposal regions with rich emotional cues.
• Utilizing spatial-temporal graph convolutional networks (GCNs) to extract reasoning features.
• Fusing region features and spatial-temporal relationship features for final emotion recognition.

Main Results:

• The proposed SE-GCN method demonstrates superior performance compared to existing approaches.
• State-of-the-art results were achieved on four challenging benchmark datasets: MHED, HEIV, VideoEmotion-8, and Ekman-6.
• The method effectively captures and utilizes spatial-temporal contextual information for accurate emotion recognition.

Conclusions:

• The SE-GCN method offers a significant advancement in video emotion recognition by effectively leveraging spatial-temporal graph convolutional networks.
• The approach provides a robust framework for analyzing emotional dynamics in videos, outperforming current benchmarks.
• Future research can explore further refinements of graph construction and feature fusion techniques.