Self-Supervised 3D Action Representation Learning With Skeleton Cloud Colorization.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Supervised learning dominates 3D skeleton-based action recognition but requires costly labeled data.
• Unlabeled skeleton data presents a challenge for effective action representation learning.

Purpose of the Study:

• To develop a self-supervised method for learning spatial-temporal representations from unlabeled 3D skeleton data.
• To advance 3D action recognition by reducing reliance on extensive manual annotation.

Main Methods:

• A novel skeleton cloud colorization technique is proposed, representing sequences as 3D point clouds and assigning artificial color labels based on spatial-temporal order.
• An auto-encoder framework with a two-stream pretraining network (fine-grained and coarse-grained colorization) is designed to learn multi-scale features.
• A Masked Skeleton Cloud Repainting task is introduced for pretraining the auto-encoder to learn informative representations.

Main Results:

• The skeleton cloud colorization approach demonstrates superior performance in unsupervised and semi-supervised 3D action recognition.
• The method achieves competitive results in fully-supervised and transfer learning settings across multiple benchmark datasets (NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, UWA3D).

Conclusions:

• Self-supervised learning via skeleton cloud colorization is effective for 3D action recognition.
• The proposed method offers a viable alternative to supervised approaches, reducing annotation burden and improving performance on unlabeled data.