Real-Time Monocular Skeleton-Based Hand Gesture Recognition Using 3D-Jointsformer

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Automatic hand gesture recognition is crucial for applications like sign language interpretation and home automation.
• Real-time performance and managing temporal dependencies are key challenges in current methods.
• Existing 3D Convolutional Neural Networks (3D-CNNs) and Transformer models have limitations in accuracy and efficiency.

Purpose of the Study:

• To develop a hybrid approach combining 3D-CNNs and Transformers for enhanced hand gesture recognition.
• To improve real-time recognition capabilities while effectively handling temporal data dependencies.
• To outperform existing state-of-the-art methods in both accuracy and processing speed.

Main Methods:

• A hybrid model integrating 3D-CNNs for semantic skeleton embedding and Transformers for long-range temporal dependency capture.
• Utilizing self-attention mechanisms within the Transformer network.
• Evaluating the model on the Briareo and Multimodal Hand Gesture datasets.

Main Results:

• Achieved high accuracy scores of 95.49% on the Briareo dataset and 97.25% on the Multimodal Hand Gesture dataset.
• Demonstrated real-time recognition performance using a standard CPU, without requiring GPUs.
• Outperformed existing methods in terms of both accuracy and processing speed.

Conclusions:

• The hybrid 3D-CNN and Transformer model offers a superior solution for real-time hand gesture recognition.
• This approach effectively addresses the challenges of local and long-range temporal dependencies.
• The method presents a significant advancement in the field, offering both high accuracy and computational efficiency.