Multi-Modal Deep Hand Sign Language Recognition in Still Images Using Restricted Boltzmann Machine

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Automatic hand sign language recognition is crucial for communication accessibility.
• Deep generative models like Restricted Boltzmann Machines (RBMs) offer potential for enhanced pattern recognition.
• Integrating multi-modal data (RGB and Depth) can improve recognition accuracy.

Purpose of the Study:

• To investigate the efficacy of Restricted Boltzmann Machines (RBMs) for automatic hand sign language recognition.
• To evaluate the performance of a multi-modal deep learning approach using RGB and Depth data.
• To assess the model's robustness against noisy input data.

Main Methods:

• A deep learning approach utilizing Restricted Boltzmann Machines (RBMs) for sign language recognition.
• Employing Convolutional Neural Networks (CNNs) for hand detection within cropped images.
• Fusing outputs from RBMs processing RGB and Depth modalities for final sign label recognition.

Main Results:

• The proposed multi-modal RBM model achieved state-of-the-art performance on multiple public datasets.
• The RBM's generative capabilities enhanced recognition of unseen hand sign data.
• The model demonstrated robustness against noisy input images.

Conclusions:

• Deep generative models, specifically RBMs, are effective for automatic hand sign language recognition.
• A multi-modal approach integrating RGB and Depth data significantly improves recognition accuracy.
• The proposed methodology offers a robust solution for real-world sign language recognition applications.