Related Experiment Video
Updated: Sep 18, 2025
06:37
Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
4.1K
Recognizing American Sign Language gestures efficiently and accurately using a hybrid transformer model
1Department of Artificial Intelligence, Faculty of Artificial Intelligence, Egyptian Russian University, Badr City, 11829, Egypt. mohammed-alysalem@eru.edu.eg.
Scientific Reports
|June 23, 2025
Summary
This study introduces a Hybrid Transformer-CNN model for efficient gesture recognition. The model achieves high accuracy and speed, outperforming existing methods in computer vision tasks like sign language interpretation.
Area of Science:
- Computer Vision
- Machine Learning
- Human-Computer Interaction
Background:
- Existing gesture recognition methods face challenges with computational demands, long-range relationship understanding, background noise sensitivity, and environmental variability.
- Convolutional Neural Networks (CNNs) capture local details but miss global context, while Vision Transformers (ViTs) model global context but are computationally expensive.
- There is a need for efficient and accurate gesture recognition models suitable for real-time applications.
Purpose of the Study:
- To propose a novel Hybrid Transformer-CNN model for gesture recognition.
- To combine the strengths of CNNs and Vision Transformers for improved performance and efficiency.
- To address the limitations of existing methods in accuracy, speed, and computational cost.
Main Methods:
- Utilized CNN layers for extracting detailed local features from hand regions.
- Employed a Vision Transformer module to capture long-range dependencies and global contextual information.
- Implemented a feature fusion strategy using element-wise multiplication for noise suppression and focus on key gesture details.
- Applied advanced data augmentation, contrastive learning, and domain adaptation for enhanced robustness.
Main Results:
- Achieved a high accuracy of 99.97% on the ASL Alphabet dataset.
- Demonstrated a processing speed of 110 frames per second.
- Required only 5.0 GFLOPs, significantly less than traditional Vision Transformer models.
- The hybrid model effectively recognizes subtle hand movements while maintaining computational efficiency.
Conclusions:
- The proposed Hybrid Transformer-CNN model offers a practical and powerful solution for gesture recognition.
- The model achieves an optimal balance between accuracy, speed, and computational efficiency.
- This work represents a significant step towards real-world applications of gesture recognition technology.
