JMM-TGT: Self-supervised 3D action recognition through joint motion masking and topology-guided transformer

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning in 3D skeleton action recognition often focuses on single motion features.
• This approach limits the capture of subtle motion variations and complex spatio-temporal relationships, leading to incomplete action understanding.

Purpose of the Study:

• To propose a novel model, the Joint Motion Masking with Topology-Guided Transformer (JMM-TGT), for enhanced 3D skeleton action recognition.
• To improve the model's ability to perceive subtle joint movements and capture complex spatio-temporal dependencies.

Main Methods:

• Implemented Joint Motion Masking (JMM) to generate masking probabilities based on joint motion differences and similarities, guiding joint masking at each time step.
• Integrated a topology-guided transformer encoder that incorporates the topological relationship between joints to adjust the attention mechanism for better spatio-temporal dependency capture.

Main Results:

• The JMM-TGT model demonstrated performance improvements ranging from 1.5% to 7.9% across different evaluation settings.
• Comparative experiments were conducted against mainstream action recognition models on NTU RGB+D 60, NTU RGB+D 120, and PUK-MMD datasets.

Conclusions:

• The proposed JMM-TGT model effectively enhances 3D skeleton action recognition by addressing limitations in existing self-supervised methods.
• The combination of joint motion masking and topology-guided attention significantly improves the understanding of complex dynamic patterns in human actions.