Bidirectional Kazakh Sign Language prosody-aware translation using computer vision and speech recognition techniques

This study presents a bidirectional communication system designed to enhance interaction between hearing-impaired and hearing individuals using gesture recognition.

The proposed framework integrates multiple components, including Kazakh Sign Language (KSL) gesture detection, closed-set sentence classification, speech recognition, and speech generation. The system is trained and evaluated using a dataset consisting of images and video recordings of KSL gestures together with audio data for Kazakh speech synthesis.

Experimental evaluation demonstrates that the overall system achieves a sentence classification accuracy of 92% across 12 closed-set sentence classes, while individual model accuracy did not fall below 87%. Compared to other approaches, the proposed method achieved competitive results in terms of accuracy on the collected dataset; however, inference speed comparisons are module-specific and measured as latency in ms/sample.

The results confirm the feasibility of the proposed approach as a proof-of-concept for improving accessibility and communication through automated sign language understanding. The system demonstrates practical applicability in bridging the communication gap between signers and non-signers, thereby promoting greater accessibility.