American Sign Language Words Recognition of Skeletal Videos Using Processed Video Driven Multi-Stacked Deep LSTM.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Dynamic sign word recognition systems face challenges with complex hand gesture interactions leading to misclassification.
• Ambiguous or similar hand motion trajectories in double-hand dynamic sign words contribute to classification errors.

Purpose of the Study:

• To propose a method for augmenting dynamic sign word feature vectors with hand dynamics knowledge.
• To improve the accuracy of sign language recognition systems by addressing classification errors in dynamic sign words.

Main Methods:

• Augmenting feature vectors of dynamic sign words with hand dynamics.
• Classifying dynamic sign words using motion patterns based on extracted and enhanced feature vectors.
• Utilizing probability density function approximation and maximal information correlation for feature enhancement.
• Employing 3D skeletal videos from a leap motion controller and feeding enhanced features as state transition patterns to a classifier.

Main Results:

• Achieved 97.98% accuracy in an experiment with 10 participants on 40 double-hand dynamic American Sign Language (ASL) words.
• Outperformed conventional methods on challenging ASL, SHREC, and LMDHG datasets by 1.47%, 1.56%, and 0.37%, respectively.

Conclusions:

• The proposed method effectively enhances dynamic sign word recognition by incorporating hand dynamics.
• The approach demonstrates superior performance over conventional methods on various datasets, indicating its robustness and potential for ubiquitous sign language recognition systems.