Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Current self-supervised methods for sign language recognition often use frame-wise learning, limiting their ability to capture rich contextual information from sign pose data.
• This leads to sub-optimal performance in accurately recognizing sign language.

Purpose of the Study:

• To propose a novel self-supervised contrastive learning framework to enhance sign language recognition.
• To extract richer contextual information using spatial-temporal consistency from sign pose data.
• To learn instance-discriminative representations for improved sign language recognition.

Main Methods:

• A self-supervised contrastive learning framework integrating spatial-temporal consistency from two perspectives.
• Utilizing both fine-grained hand and coarse-grained trunk information, encoding them into latent spaces, and constraining feature consistency.
• Introducing first-order motion information and bridging interaction between motion and joint modalities for bidirectional knowledge transfer.

Main Results:

• The proposed method achieves new state-of-the-art performance on four public benchmarks for sign language recognition.
• Demonstrates a notable margin of improvement compared to existing methods.
• The framework effectively excavates rich context and learns discriminative representations.

Conclusions:

• The proposed self-supervised contrastive learning framework is effective for sign language recognition.
• Integrating multi-granularity features (hands, trunk) and multi-modalities (motion, joints) significantly enhances representation learning.
• The method offers a simple yet powerful approach to advance the field of sign language recognition.