Generative Sign-Description Prompts with Multi-Positive Contrastive Learning for Sign Language Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Linguistics

Background:

• Sign language recognition systems struggle with multimodal annotation for hierarchical semantics.
• Current methods lack the ability to capture both sequential hand motions and concurrent non-manual cues effectively.

Purpose of the Study:

• To propose GSP-MC, the first method integrating generative large language models (LLMs) into sign language recognition.
• To develop a system capable of precise multipart descriptions and hierarchical semantic understanding of sign language.

Main Methods:

• Utilized retrieval-augmented generation with domain-specific LLMs and expert-validated corpora.
• Employed a dual-encoder architecture for bidirectional alignment of skeleton features with multi-level text descriptions (global, synonym, part) via probabilistic matching.
• Combined global and part-level losses with KL divergence optimization for robust alignment and semantic capture.

Main Results:

• Achieved state-of-the-art accuracy: 97.1% on Chinese SLR500 and 97.07% on Turkish AUTSL.
• Demonstrated superior performance compared to existing methods (SSRL and SML).
• Confirmed the cross-lingual potential of the GSP-MC approach.

Conclusions:

• GSP-MC effectively bridges the gap in multimodal annotation for sign language recognition.
• The method accurately captures sign semantics and detailed dynamics, enabling robust recognition.
• This work advances inclusive communication technologies through improved sign language understanding.