Hierarchical intertwined graph representation learning for skeleton-based action recognition

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Graph Convolutional Networks (GCNs) excel at human skeleton-based action recognition by modeling skeletal joints as graphs.
• Existing GCNs often focus on single-frame spatial context or holistic temporal sequences, neglecting multi-scale spatial-temporal interactions.
• This limitation impacts the understanding of complex actions with varying temporal dynamics.

Purpose of the Study:

• To propose a novel Hierarchical Intertwined Graph Learning Framework (HI-GCN) for improved skeleton-based action recognition.
• To address the limitations of existing GCNs in capturing multi-scale spatial-temporal relationships.
• To enhance the recognition of complex actions by integrating diverse temporal scales.

Main Methods:

• Developed the Hierarchical Intertwined Graph Learning Framework (HI-GCN).
• Introduced an Intertwined Context Graph Convolution module for integrating spatial-temporal information across adjacent frames at various temporal scales.
• Incorporated a Shifted Window Temporal Transformer module with multi-scale receptive fields for advanced temporal dependency modeling.

Main Results:

• HI-GCN achieved superior performance on multiple benchmark datasets for skeleton-based action recognition.
• Achieved 93.3% accuracy on NTU RGB+D 60 (cross-subject).
• Achieved 90.3% accuracy on NTU RGB+D 120 (cross-subject) and 97.0% on NW-UCLA.

Conclusions:

• The proposed HI-GCN effectively captures subtle topological variations and refines spatial-temporal representations.
• The framework demonstrates significant improvements over state-of-the-art methods in skeleton-based action recognition.
• HI-GCN offers a more comprehensive approach to understanding complex human actions from skeletal data.