Recognition of Fingerspelling Sequences in Polish Sign Language Using Point Clouds Obtained from Depth Images

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Pattern Recognition

Background:

• Sign language recognition is crucial for communication accessibility.
• Existing methods often struggle with the dynamic and coarticulated nature of finger spelling.
• Efficiently recognizing sequences of static letters in dynamic motion presents a significant challenge.

Purpose of the Study:

• To develop and evaluate a robust method for recognizing sequences of static letters in the Polish finger alphabet.
• To explore the effectiveness of various point cloud descriptors and hidden Markov model architectures for classifying dynamic hand gestures.
• To assess the generalizability of the proposed hand shape representation for single-frame hand posture recognition.

Main Methods:

• Utilized point cloud descriptors including viewpoint feature histogram, eigenvalues-based descriptors, ensemble of shape functions, and global radius-based surface descriptor.
• Employed networks of hidden Markov models (HMMs) trained on transitions between letter postures for sequence classification.
• Investigated three types of left-to-right HMMs and two dictionary-dependent/independent network architectures.

Main Results:

• The method demonstrated successful recognition of Polish finger alphabet sequences using diverse point cloud descriptors and HMM configurations.
• The hand shape representation proved effective for recognizing individual hand postures in single frames, validated on a large American finger alphabet dataset.
• Achieved high accuracy in classifying dynamic, coarticulated finger movements.

Conclusions:

• The proposed method offers a promising approach for automated sign language recognition, particularly for dynamic finger spelling.
• The integration of advanced point cloud descriptors with HMMs effectively captures the nuances of coarticulated hand gestures.
• The system's adaptability to single-frame posture recognition highlights its versatility for broader hand gesture analysis applications.