Bidirectional Kazakh Sign Language prosody-aware translation using computer vision and speech recognition techniques

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Linguistics

Background:

• Effective communication between hearing-impaired and hearing individuals remains a significant challenge.
• Existing assistive technologies often lack comprehensive bidirectional capabilities.
• Automated sign language recognition is crucial for bridging this communication gap.

Purpose of the Study:

• To develop and evaluate a bidirectional communication system integrating Kazakh Sign Language (KSL) recognition and speech synthesis.
• To enhance interaction and accessibility for deaf and hearing individuals.
• To demonstrate a proof-of-concept for automated sign language understanding.

Main Methods:

• Integration of KSL gesture detection, closed-set sentence classification, speech recognition, and speech generation.
• Training and evaluation using a dataset of KSL gestures (images/video) and Kazakh speech audio.
• Utilizing machine learning models for gesture recognition and speech synthesis.

Main Results:

• The overall system achieved 92% accuracy in classifying 12 closed-set KSL sentences.
• Individual model accuracies were consistently above 87%.
• The system demonstrated competitive accuracy compared to existing approaches.

Conclusions:

• The proposed system is a feasible proof-of-concept for improving communication accessibility.
• Automated sign language understanding can effectively bridge the communication gap.
• The technology holds practical applicability for signers and non-signers.