American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose Estimation

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Sign language recognition is crucial for the deaf and hard of hearing community.
• Vision-based approaches offer cost-effectiveness despite historical accuracy challenges compared to sensor-based methods.

Purpose of the Study:

• To develop and evaluate a cost-effective system for recognizing American sign language characters using readily available web cameras.
• To explore the efficacy of hand joint features derived from RGB images for sign language recognition.

Main Methods:

• Utilized the MediaPipe Hands algorithm to estimate hand joint coordinates from webcam-captured RGB images.
• Extracted features including joint distances and angles between vectors and 3D axes.
• Employed Support Vector Machine (SVM) and Light Gradient Boosting Machine (GBM) classifiers for character recognition.
• Evaluated performance on the ASL Alphabet, Massey, and Finger Spelling A datasets.

Main Results:

• Achieved high recognition accuracies: 99.39% on the Massey dataset, 98.45% on the Finger Spelling A dataset, and 87.60% on the ASL Alphabet dataset.
• The proposed vision-based system demonstrated superior performance compared to previous studies.

Conclusions:

• The developed American sign language recognition system is cost-effective, computationally inexpensive, and requires no specialized hardware.
• This approach offers a practical and accurate solution for facilitating communication for individuals who are deaf or hard of hearing.