Two-stream spatio-temporal GCN-transformer networks for skeleton-based action recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Prior skeleton-based action recognition methods often use serial GCNs and attention, neglecting crucial indirect skeletal relationships.
• This limitation hinders accuracy by treating the skeleton as isolated, overlooking correlated yet indirectly connected parts.

Purpose of the Study:

• To propose a novel architecture, SA-TDGFormer, that overcomes limitations of serial approaches for accurate skeleton-based action recognition.
• To enhance motion information encoding by effectively integrating local and global spatio-temporal features.

Main Methods:

• Introduced a parallel architecture combining Graph Convolutional Networks (GCNs) and Transformer models (SA-TDGFormer).
• Employed a dual-stream structure: a spatiotemporal GCN stream for topological and motion representations, and a spatiotemporal Transformer stream for global inter-joint relationships.
• Utilized a late fusion strategy to merge complementary features from both streams, enriching action representations.

Main Results:

• Empirical validation on NTU RGB+D 60, NTU RGB+D 120, and Kinetics-Skeleton datasets demonstrated model effectiveness.
• Achieved 1-5% improvement in human action recognition accuracy on the NTU RGB+D 60 dataset compared to existing frameworks.
• The parallel structure effectively captures both local and global spatio-temporal features, leading to superior recognition performance.

Conclusions:

• The proposed SA-TDGFormer model significantly improves skeleton-based action recognition accuracy.
• The parallel GCN and Transformer architecture effectively addresses limitations of previous serial methods.
• This approach offers a more robust and accurate method for encoding human motion dynamics.