An explainable hybrid CNN-transformer model for sign language recognition on edge devices using adaptive fusion and...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Deep learning (DL) for sign language recognition (SLR) faces challenges with monolingual data, interpretability, and computational intensity for edge deployment.
• There is a growing need for inclusive, real-time communication technologies, highlighting the importance of efficient and deployable SLR systems.

Purpose of the Study:

• To present TinyMSLR, an explainable, lightweight framework for isolated-sign classification on resource-constrained devices.
• To improve accuracy and efficiency for multilingual SLR through a novel dual-teacher knowledge distillation (KD) scheme.

Main Methods:

• TinyMSLR combines ConvNeXt-Tiny and Swin Transformer encoders with an adaptive fusion gate.
• A dual-teacher knowledge distillation (KD) scheme transfers knowledge from large models to a compact student model.
• Evaluation was conducted on multilingual datasets (DGS-2014T, CSL) focusing on isolated-sign recognition.

Main Results:

• TinyMSLR achieved high accuracy: 99.28% training, 99.01% validation, and 98.96% F1-score.
• The model is lightweight, with under 2.7M parameters.
• Inference latency is 24 ms on CPUs and under 13.5 ms on edge GPUs.

Conclusions:

• TinyMSLR offers a practical balance of accuracy, efficiency, and explainability for multilingual isolated-sign recognition.
• The framework is well-suited for deployment on edge devices, enabling real-time communication.
• This work advances the development of inclusive communication technologies for the deaf and hard-of-hearing community.