Skeleton-Based Emotion Recognition Based on Two-Stream Self-Attention Enhanced Spatial-Temporal Graph Convolutional...

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Emotion recognition is a growing research area, yet the gesture modality is underutilized due to limited labeled 3D skeleton data.
• Existing action recognition models use graph neural networks for joint connections, but this approach is unexplored in gesture-based emotion recognition.

Purpose of the Study:

• To address the scarcity of labeled 3D skeleton data for emotion recognition.
• To propose and evaluate a novel self-attention enhanced spatial-temporal graph convolutional network (GCN) for skeleton-based emotion recognition.

Main Methods:

• Utilized a pose estimation method to extract 3D skeleton coordinates from the IEMOCAP database.
• Developed a GCN model incorporating self-attention mechanisms to dynamically model joint relationships and spatial-temporal information.
• Compared the proposed model against existing methods for skeleton-based emotion recognition.

Main Results:

• The proposed self-attention enhanced GCN model significantly outperformed baseline models in emotion recognition accuracy.
• Extracted skeleton data features demonstrably enhanced the performance of multimodal emotion recognition systems.
• The self-attention mechanism effectively captured dynamic joint connections, providing supplementary emotional expression information.

Conclusions:

• Skeleton-based analysis using advanced GCNs is a promising direction for emotion recognition.
• The proposed model offers a robust solution for leveraging gesture modality in affective computing.
• Integrating dynamic skeletal features can substantially improve the accuracy and robustness of emotion recognition systems.