American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture System

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Sign language recognition faces challenges due to limited datasets and complex data acquisition.
• Wearable inertial motion capture systems offer a potential solution for capturing sign language data.

Purpose of the Study:

• To create a comprehensive American Sign Language (ASL) dataset using wearable inertial motion capture.
• To develop and evaluate deep learning models for ASL sentence recognition and end-to-end translation.

Main Methods:

• Collected a dataset of 300 common ASL sentences from three volunteers using a wearable inertial motion capture system.
• Designed a recognition network combining convolutional neural networks, bi-directional long short-term memory, and connectionist temporal classification.
• Developed an encoder-decoder model based on long short-term memory with global attention for translation.

Main Results:

• The recognition model achieved 99.07% accuracy at the word level and 97.34% at the sentence level.
• The end-to-end translation model achieved a word error rate of 16.63%.

Conclusions:

• The proposed method demonstrates the potential for accurate ASL recognition and translation using inertial sensor data.
• This approach can enhance communication accessibility for the deaf community by enabling reliable sign language interpretation.