Modeling the Internal and Contextual Attention for Self-Supervised Skeleton-Based Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Multimodal contrastive learning excels in self-supervised skeleton-based action recognition.
• Existing methods suffer from modality imbalance, hindering alignment and feature representation.
• This leads to confusion between modalities and weakened performance in action recognition.

Purpose of the Study:

• To overcome modality imbalance in skeleton-based action recognition.
• To explore intra-modality feature-wise self-similarity and inter-modality instance-wise cross-consistency.
• To develop a novel language-skeleton contrastive learning framework for enhanced action recognition.

Main Methods:

• Proposed a novel language-skeleton contrastive learning framework (MICA).
• Introduced Feature Modulation to create a skeleton-language action conceptual domain.
• Developed Frequency Feature Learning with a Frequency-domain Spatial-Temporal block (FreST) focusing on key joints in the frequency domain.

Main Results:

• Achieved remarkable action recognition performance on NTU RGB+D 60 and NTU RGB+D 120 datasets.
• Demonstrated significant improvements on the PKU-MMD dataset, outperforming classical methods by at least 4.6%.
• Effectively captured internal and contextual attention information for robust action recognition.

Conclusions:

• The proposed MICA framework effectively addresses modality imbalance in skeleton-based action recognition.
• Leveraging global perspective and focus adaptation through frequency domain analysis enhances feature representations.
• MICA offers a promising approach for advancing self-supervised skeleton-based action recognition.