Novel Spatio-Temporal Continuous Sign Language Recognition Using an Attentive Multi-Feature Network.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Continuous Sign Language Recognition (CSLR) faces challenges due to unsegmented signs in video streams.
• Existing deep learning methods often rely solely on RGB features (full-frame, hands, face), limiting multi-feature learning.
• Information scarcity and frame-level noise in CSLR training data can hinder performance.

Purpose of the Study:

• To propose a novel spatio-temporal continuous sign language recognition method.
• To enhance CSLR by integrating extra keypoint features into an attentive multi-feature network.
• To improve the accuracy and robustness of sign language recognition systems.

Main Methods:

• Developed a novel attentive multi-feature network for spatio-temporal CSLR.
• Incorporated extra keypoint features to enrich the input data for CSLR.
• Utilized attention layers in spatial and temporal modules to emphasize crucial features simultaneously.
• Evaluated the proposed method on CSL and PHOENIX CSLR datasets.

Main Results:

• The proposed method achieved superior performance compared to state-of-the-art methods.
• Demonstrated significant improvements in Word Error Rate (WER) on both CSL and PHOENIX datasets.
• Achieved WER score reductions of 0.76 on CSL and 20.56 on PHOENIX datasets.

Conclusions:

• The attentive multi-feature network effectively enhances continuous sign language recognition.
• Integrating keypoint features and employing attention mechanisms improves CSLR accuracy.
• The proposed method represents a significant advancement in the field of sign language recognition.