Multi-cue temporal modeling for skeleton-based sign language recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Linguistics

Background:

• Sign languages are visual-gestural languages crucial for the Deaf community.
• Sign Language Recognition (SLR) typically focuses on manual features, potentially overlooking other vital articulators.
• Facial expressions and body movements significantly contribute to the meaning and context of signs.

Purpose of the Study:

• To develop an advanced Sign Language Recognition (SLR) framework.
• To exploit multi-articulatory information (body, hands, face) for improved sign gloss recognition.
• To analyze the contribution of individual visual cues to SLR performance.

Main Methods:

• Proposed a framework combining Spatial-Temporal Graph Convolutional Networks (ST-GCNs) and Multi-Cue Long Short-Term Memorys (MC-LSTMs).
• Utilized ST-GCNs for learning representations from upper body and hand movements.
• Employed pre-trained Convolutional Neural Networks (CNNs) to extract spatial embeddings for hand shape and facial expressions.

Main Results:

• Achieved comparable recognition performance to state-of-the-art skeleton-based methods.
• Demonstrated that incorporating multiple visual cues (multi-articulatory) enhances recognition accuracy.
• Observed significant improvements in recognizing sign classes where multi-cue information is critical.

Conclusions:

• The proposed multi-cue SLR framework effectively integrates diverse articulatory information.
• Combining manual, body, and facial cues offers superior performance over methods relying solely on manual features.
• This approach provides valuable insights into the linguistic importance of different visual cues in Sign Language.