MSST-RT: Multi-Stream Spatial-Temporal Relative Transformer for Skeleton-Based Action Recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Skeleton-based human action recognition leverages graph convolution networks (GCNs), notably ST-GCN, for spatial-temporal pattern learning.
• Existing GCN methods have limitations in capturing long-range dependencies due to restricted receptive fields, crucial for accurate action recognition.

Purpose of the Study:

• To address the limitations of short-range correlation learning in GCNs for action recognition.
• To introduce a novel spatial-temporal relative transformer (ST-RT) model capable of capturing long-range spatial-temporal dependencies.

Main Methods:

• Proposed the spatial-temporal relative transformer (ST-RT) incorporating relay nodes to overcome transformer architecture limitations and break inherent skeleton topology and sequence order constraints.
• Integrated multi-scale dynamic motion information extraction.
• Developed a multi-stream spatial-temporal relative transformer (MSST-RT) by fusing four ST-RTs processing different skeleton sequence types.

Main Results:

• The MSST-RT model demonstrated competitive performance against state-of-the-art (SOTA) methods.
• Extensive experiments on NTU RGB+D, NTU RGB+D 120, and UAV-Human datasets validated the proposed approach's effectiveness.

Conclusions:

• The proposed MSST-RT effectively captures essential long-range spatial-temporal dependencies for human action recognition.
• The novel ST-RT architecture offers a promising direction for advancing skeleton-based action recognition research.