Language-Guided 3-D Action Feature Learning Without Ground-Truth Sample Class Label.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• 3-D action recognition from point cloud sequences lags behind skeleton-based methods.
• Leveraging cross-modality supervision is a promising direction to bridge this gap.

Purpose of the Study:

• To introduce text's weak supervision for self-supervised 3-D action feature learning (S3AFL) from point cloud sequences.
• To narrow the performance disparity between point cloud and skeleton-based 3-D action recognition.

Main Methods:

• Utilized RGB-point cloud pairs with text generated from RGB via image captioning for weak supervision.
• Employed cross- and intra-modality contrastive learning (CL) with multistage semantic refinement.
• Introduced multirank max-pooling (MR-MP) for enhanced point set feature representation.

Main Results:

• Achieved performance gains of up to 10.8% on NTU RGB+D 60, 10.4% on NTU RGB+D 120, and 8.0% on N-UCLA.
• Significantly reduced the performance gap between point cloud sequence and skeleton-based action recognition methods.
• Demonstrated the generality of text-based weak supervision transfer to skeleton-based methods.

Conclusions:

• Text-based weak supervision effectively enhances self-supervised 3-D action feature learning from point cloud sequences.
• The proposed S3AFL framework with MR-MP offers a robust approach for fine-grained action recognition.
• The cross-modality transfer learning strategy shows broad applicability and potential for future research in human action analysis.