Video-Based Arabic Sign Language Recognition with Mediapipe and Deep Learning Techniques

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Deaf and hearing-impaired individuals in the Arab world face significant communication barriers.
• Existing solutions for sign language recognition are often inaccessible or require specialized hardware.
• There is a need for an affordable and widely accessible Arabic Sign Language (ArSL) recognition system.

Purpose of the Study:

• To develop an affordable, video-based ArSL recognition system using standard cameras.
• To enhance communication accessibility for the deaf and hearing-impaired community in the Arab world.
• To achieve high accuracy in recognizing full Arabic words through sign language.

Main Methods:

• Utilized Mediapipe for real-time extraction of hand, face, and pose landmarks from video streams.
• Developed a hybrid deep learning model integrating Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) layers.
• Employed a CNN-BiLSTM architecture for comprehensive spatiotemporal feature extraction to differentiate complex signs.

Main Results:

• Achieved 96% overall accuracy on the JUST-SL dataset.
• Attained an impressive 99% accuracy on the KArSL dataset.
• Demonstrated superior accuracy compared to previous research, especially for full Arabic word recognition.

Conclusions:

• The developed ArSL recognition system significantly enhances communication accessibility.
• The system's high accuracy and affordability make it a valuable tool for the deaf and hearing-impaired community.
• This research represents a significant advancement in sign language recognition technology for the Arab world.