Video-Based Sign Language Recognition via ResNet and LSTM Network.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Sign language recognition technology aids communication for hearing-impaired individuals.
• Deep learning offers technical support for sign language recognition.
• Traditional methods using convolutional neural networks face challenges in feature extraction and computational resources.

Purpose of the Study:

• To present a novel video-based sign language recognition method.
• To improve the accuracy and efficiency of sign language recognition.
• To address the limitations of traditional methods in feature extraction and computational demands.

Main Methods:

• Utilized a Residual Network (ResNet) as the backbone for feature extraction.
• Employed Long Short-Term Memory (LSTM) networks to capture long sequence features.
• Combined ResNet for spatial feature extraction and LSTM for temporal feature learning from sign language videos.

Main Results:

• The proposed ResNet-LSTM method achieved an accuracy of 86.25% on the Argentine Sign Language (LSA64) dataset.
• Demonstrated superior performance with an F1-score of 84.98% and precision of 87.77%.
• Effectively extracted spatio-temporal features, significantly improving sign language action recognition rates.

Conclusions:

• The ResNet-LSTM method offers an effective solution for video-based sign language recognition.
• The approach enhances the extraction of spatio-temporal features, leading to improved recognition accuracy.
• This technology holds promise for better communication tools for the hearing-impaired community.