Lip reading systems for Urdu alphabets in diverse environments

Lip reading technology has potential use across various fields, significantly enhancing communication for the deaf, aiding in noisy settings, and supporting information security through silent password entry. Although, notable progress has been made in constructing datasets of different types like digits, alphabets, words, phrases, and sentences levels for lip reading in various languages. However, developing a robust Urdu lip reading model remains a challenge due to the lack of a suitable dataset. Moreover, difficulties in adapting previous models, such as the LipNet model to Urdu. To address these barriers, we present the ULRA (Urdu lip reading alphabets) dataset, leverage advanced data augmentation techniques, and evaluate three cutting-edge DNN models: a LipNet-based 2D-CNN model, a Hybrid 2D_3D-CNN model, and a baseline 3D-CNN model. Each model undergoes rigorous testing in diverse environments, with both familiar and unfamiliar data. The results reveal that the LipNet-based 2D-CNN model achieves an impressive 81.97% accuracy on unknown data across diverse environments, while the Hybrid model excels in generalization, reaching 69.45% accuracy on unfamiliar data, thanks to its superior spatiotemporal feature extraction capabilities. Additionally, precision, recall, and F1-Score values of LipNet-Based 2D CNN are 0.83, 0.82, and 0.82 respectively. All three values of this model are also higher than the other two models. These findings underscore the strengths of various DNN architectures and the critical advancements made possible by the ULRA dataset, paving the way for future breakthroughs in Urdu lip reading research.