Lip reading systems for Urdu alphabets in diverse environments

Area of Science:

• Computer Science
• Artificial Intelligence
• Speech and Language Processing

Background:

• Lip reading technology offers diverse applications, including aiding the deaf and enhancing security.
• Existing lip reading datasets and models face limitations, particularly for under-resourced languages like Urdu.
• Developing robust Urdu lip reading systems is hindered by data scarcity and model adaptability challenges.

Purpose of the Study:

• To introduce the Urdu Lip Reading Alphabets (ULRA) dataset for Urdu lip reading research.
• To evaluate the performance of deep neural network (DNN) models for Urdu lip reading.
• To address the challenges in Urdu lip reading model development and dataset creation.

Main Methods:

• Creation of the ULRA dataset, specifically for Urdu lip reading alphabets.
• Application of advanced data augmentation techniques to enhance dataset robustness.
• Evaluation of three DNN models: LipNet-based 2D-CNN, Hybrid 2D_3D-CNN, and a baseline 3D-CNN.

Main Results:

• The LipNet-based 2D-CNN model achieved 81.97% accuracy on unfamiliar Urdu data across diverse environments.
• The Hybrid 2D_3D-CNN model demonstrated strong generalization with 69.45% accuracy on unfamiliar data.
• The LipNet-based 2D-CNN model outperformed others with precision, recall, and F1-scores of 0.83, 0.82, and 0.82, respectively.

Conclusions:

• The ULRA dataset is a significant contribution to Urdu lip reading research.
• Different DNN architectures offer varying strengths for lip reading tasks.
• The developed models and dataset pave the way for improved Urdu lip reading technology.