Pose-Based Static Sign Language Recognition with Deep Learning for Turkish, Arabic, and American Sign Languages

Area of Science:

• Artificial Intelligence
• Computer Vision
• Computational Linguistics

Background:

• Artificial intelligence advancements improve communication for hearing-impaired individuals.
• Cross-lingual Sign Language Recognition (SLR) systems are crucial for bridging communication gaps.
• Existing SLR frameworks often lack robustness across diverse languages.

Purpose of the Study:

• To present a robust cross-lingual Sign Language Recognition (SLR) framework for Turkish, American English, and Arabic.
• To compare the efficacy of different deep learning architectures for SLR.
• To provide insights into model generalization for pose-based SLR systems.

Main Methods:

• Utilized the MediaPipe library for efficient hand landmark extraction.
• Constructed diverse datasets from nine public-domain sources for Turkish, American English, and Arabic sign languages.
• Conducted a comparative evaluation of ConvNeXt (CNN-based), Swin Transformer (ViT-based), and Vision Mamba (SSM-based) architectures.

Main Results:

• Vision Transformers and state space models demonstrated superior performance in capturing spatial cues.
• The framework achieved stable and consistent feature representation across languages.
• Comparative analysis highlighted model generalization capabilities across distinct sign languages.

Conclusions:

• Contemporary vision Transformers and state space models are highly effective for cross-lingual SLR.
• The study offers valuable insights for selecting appropriate models in pose-based SLR systems.
• The developed framework enhances communication accessibility for individuals with hearing impairments.