SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language Understanding.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Sign language understanding (SLU) methods face challenges like overfitting and limited interpretability due to data scarcity.
• Hand gestures are fundamental to sign language expression, yet current models struggle to fully leverage this information.

Purpose of the Study:

• To propose the first self-supervised pre-trainable framework, SignBERT+, for enhanced sign language understanding.
• To address data limitations and improve model interpretability in sign language recognition and translation.

Main Methods:

• Developed the SignBERT+ framework incorporating model-aware hand prior, treating hand pose as embedded visual tokens.
• Employed multi-level masked modeling strategies (joint, frame, clip) for self-supervised pre-training on existing sign data.
• Designed effective prediction heads for downstream tasks like isolated and continuous sign language recognition (SLR) and sign language translation (SLT).

Main Results:

• SignBERT+ achieved state-of-the-art performance across three main SLU tasks.
• The framework demonstrated significant gains in accuracy and effectiveness compared to existing methods.
• Self-supervised pre-training effectively modeled sign data statistics and improved generalization.

Conclusions:

• The proposed SignBERT+ framework offers a robust solution for sign language understanding, overcoming common limitations.
• Incorporating hand pose as visual tokens and employing self-supervised learning are key to achieving superior performance.
• This work advances the field of SLU, paving the way for more accessible and accurate sign language technologies.