Sensor Fusion of Motion-Based Sign Language Interpretation with Deep Learning

Area of Science:

• Engineering
• Computer Science
• Human-Computer Interaction

Background:

• Sign language is crucial for hearing-impaired individuals, but societal knowledge is limited, creating communication barriers.
• Existing computer vision (CV) methods for sign language recognition face limitations due to visual angle dependency and environmental factors.
• CV-based systems often require expert teams and expensive hardware, increasing real-world application costs.

Purpose of the Study:

• To design and implement a smart wearable American Sign Language (ASL) interpretation system.
• To overcome the limitations of CV-based approaches by utilizing sensor fusion and deep learning.
• To provide a cost-effective and accessible solution for facilitating communication for the hearing-impaired community.

Main Methods:

• Developed a deep learning model incorporating sensor fusion of six inertial measurement units (IMUs).
• Attached IMUs to fingertips and the back of the hand to capture gesture data.
• The system is designed to be independent of the field of view, unlike CV methods.

Main Results:

• Achieved an average recognition rate of 99.81% for dynamic ASL gestures.
• The wearable system demonstrated high accuracy and robustness in recognizing sign language.
• The proposed method is not restricted by visual angle or environmental conditions.

Conclusions:

• The developed wearable ASL interpretation system offers a highly accurate and reliable solution.
• Integration with Information and Communication Technology (ICT) and Internet of Things (IoT) can further enhance its utility.
• This technology has the potential to significantly improve communication and quality of life for the hearing-impaired.