Dynamic Semantic-based Spatial-Temporal Graph Convolution Network for Skeleton-based Human Action Recognition.

Area of Science:

• Computer Vision
• Image Processing
• Artificial Intelligence

Background:

• Skeleton-based human action recognition is crucial in computer vision.
• Graph convolutional networks (GCNs) show promise but often overlook joint/edge types and temporal order.
• Existing GCNs struggle to capture intrinsic semantic information for accurate action recognition.

Purpose of the Study:

• To propose a novel dynamic semantic-based spatial-temporal graph convolution network (DS-STGCN).
• To address limitations in GCNs by incorporating joint/edge types and frame order for improved action recognition.
• To enhance the representation of intrinsic semantic information in skeleton-based action recognition.

Main Methods:

• Developed DS-STGCN with two dynamic semantic modules for spatial and temporal contexts.
• Implicitly encoded joint and edge types within the spatial module.
• Implicitly encoded the occurrence order of frames within the temporal module.

Main Results:

• DS-STGCN achieved consistent performance improvements across various backbones.
• The model significantly outperformed state-of-the-art methods on NTU-RGB+D 60(120), Kinetics-400, and FineGYM datasets.
• Notable outperformance was observed on the challenging Kinetics-400 dataset compared to other GCN-based methods.

Conclusions:

• The proposed dynamic semantic modules enhance GCN performance in action recognition.
• DS-STGCN offers a superior approach to skeleton-based human action recognition.
• The method demonstrates significant advancements over existing state-of-the-art techniques.