Sign Pulse: a hybrid AI-based learning framework for enhancing education among deaf and hard-of-hearing students

Sign Pulse is a hybrid AI-driven educational framework designed to enhance STEM learning for deaf and hard-of-hearing (DHH) students in grades 1-4 (approximately 6-10 years of age) through native sign language interaction. Addressing the limited availability for low-resource sign languages, the system is among the first to focus on Palestinian Sign Language (PSL) within a classroom-ready learning environment.

Sign Pulse integrates real-time sign language recognition with intelligent pedagogical feedback by combining a Siamese Vision Transformer and a β-Variational Autoencoder (β-VAE) for robust spatial-temporal gesture modelling, together with Large Language Models (LLMs) augmented through Retrieval-Augmented Generation (RAG) to deliver curriculum-aligned explanations, questions, and feedback. The system was evaluated on a newly developed PSL dataset consisting of 434 annotated classroom video samples covering foundational STEM concepts.

Experimental results demonstrate strong recognition and verification performance, achieving a Top-1 accuracy of 90%, Top-5 accuracy of 99.65%, an AUC of 92%, and an equal error rate (EER) of 7.8%. Beyond technical performance, the integration of RAG-enhanced LLMs significantly improved the quality of educational feedback, yielding a BERT Score F1 of 88% and a ROUGE-Recall of 55%, substantially outperforming baseline fixed-response systems. These results demonstrate that Sign Pulse effectively transforms sign language recognition from a standalone accessibility tool into an interactive learning system that supports personalised, concept-driven STEM education. By unifying low-resource sign language data, advanced vision-based recognition, and generative pedagogical reasoning within a single framework, Sign Pulse offers a scalable and replicable model for inclusive AI-powered education for DHH learners in underrepresented linguistic communities.