A dual-stream deep learning framework for continuous sign language recognition to enhance communication accessibility...

Human-Computer Interaction (HCI) depends critically on hand gesture recognition, which also provides the basis of sign language communication for those with hearing problems. In automatic sign language identification systems, however, the contribution of non-manual elements like head posture and facial emotions has sometimes been disregarded. In order to fully capture the diversity of sign language modalities and to enhance the quality of life of deaf people in the region of Ha'il, we present in this study a dual-stream convolutional neural network (TS-CNN) that separately models hand motions and head positions. We provide fusion techniques that improve feature correlation and representation in order to guarantee efficient cooperation between the two streams. Moreover, we use a Feature Enhancement Module (FEM) to enhance gloss-level prediction accuracy and fine temporal alignment. Two benchmark datasets, RWTH-PHOENIX-Weather 2014 and CSL Split II, are used to assess the TS-CNN framework and show a notable increase in recognition performance. Competitive Word Error Rates (WER) obtained by TS-CNN from experimental data validate its performance in simulating the complicated multimodal character of continuous sign language recognition.