Real-Time Sign Language Interpretation via Customized Sign Language Gloves and Motion Retrieval

Area of Science:

• Human-Computer Interaction
• Assistive Technology
• Sign Language Recognition

Background:

• Existing sign language interpretation systems face challenges like occlusion and high computational costs.
• Vision-based methods struggle with environmental variations, while Deep Neural Network (DNN) approaches are too resource-intensive for mobile use.
• There's a need for efficient, portable sign language interpretation solutions.

Purpose of the Study:

• To develop a real-time sign language interpretation system using sign language gloves and a lightweight motion retrieval method.
• To enable communication between signers and non-signers on resource-constrained platforms like mobile devices and embedded systems.
• To overcome the limitations of current vision-based and DNN-based interpretation systems.

Main Methods:

• Sign language gloves equipped with flex sensors, IMU, and pressure sensors were developed to capture detailed gesture data.
• A comprehensive gesture dataset was created, with feature analysis to reduce redundancy.
• A motion retrieval method utilizing feature labeling and a gesture retrieval algorithm was implemented for efficient interpretation.

Main Results:

• The proposed system achieved an average recognition accuracy of 92% for 300 sign language words.
• The motion retrieval method demonstrated low computational complexity and a suitable data structure for embedded systems.
• The system proved effective for real-time performance and portability.

Conclusions:

• The developed sign language gloves and lightweight motion retrieval method offer a viable solution for real-time interpretation on mobile and embedded systems.
• The system effectively addresses the limitations of existing approaches, providing high accuracy and portability.
• This technology has the potential to significantly improve communication accessibility for the deaf and hard-of-hearing community.