Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Word-level sign language recognition (WSLR) is crucial for continuous sign language recognition (CSLR).
• Accurate gloss prediction and boundary detection remain significant challenges in WSLR.
• Existing automated feature extraction methods are computationally expensive and less accurate.

Purpose of the Study:

• To enhance gloss prediction accuracy in WSLR.
• To reduce the time and computational overhead associated with gloss prediction.
• To develop a systematic approach for gloss prediction using a transformer model.

Main Methods:

• Proposed a Sign2Pose Gloss prediction transformer model for WSLR.
• Implemented a modified key frame extraction technique using histogram difference and Euclidean distance.
• Utilized pose vector augmentation with perspective transformation and joint angle rotation.
• Employed YOLOv3 for signing space detection and hand gesture tracking for normalization.

Main Results:

• Achieved top 1% recognition accuracy of 80.9% on WLASL100 and 64.21% on WLASL300.
• Demonstrated superior performance compared to state-of-the-art approaches.
• Observed a 17% performance improvement on the WLASL 100 dataset.
• YOLOv3 integration improved accuracy and prevented model overfitting.

Conclusions:

• The proposed systematic approach significantly enhances WSLR gloss prediction accuracy.
• The integration of key frame extraction, augmentation, and pose estimation improves model precision.
• The Sign2Pose model offers a computationally efficient and accurate solution for sign language recognition.