Related Experiment Video
Updated: Oct 5, 2025
08:15
Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision
Published on: March 28, 2025
811
Development of a low-resource wearable continuous gesture-to-speech conversion system.
Vijayalakshmi Parthasarathy1, Nagarajan Thangavelu2, Jayapriya Ramesh1
1Speech Lab, Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, India.
Disability and Rehabilitation. Assistive Technology
|January 21, 2022
Summary
This study introduces a wearable device that converts sign language gestures into speech using motion sensors and Hidden Markov Models (HMMs). The system aims to improve communication for the hearing impaired by enabling real-time gesture-to-speech conversion.
Area of Science:
- Assistive Technology
- Human-Computer Interaction
- Biomedical Engineering
Background:
- Hearing impairment significantly limits communication and daily interaction.
- Sign language, while primary for the deaf community, faces challenges in efficacy and standardization, especially in India.
- Existing assistive devices often lack the necessary features for seamless communication between hearing and hearing-impaired individuals.
Purpose of the Study:
- To develop a compact, low-resource, wireless, motion sensor-based module for single and double hand-gesture recognition.
- To facilitate communication between unimpaired and hearing-impaired populations by converting sign language into speech.
- To address the lack of standardized sign language datasets by creating a customizable gesture recognition system.
Main Methods:
- A two-step process involving Hidden Markov Model (HMM) based gesture-to-text conversion and bilingual text-to-speech synthesis.
- Implementation of multi-threading for parallel processing to minimize system delay.
- Modeling and testing of continuous gesture recognition using ergodic HMMs for non-gesture hand motions, applied to American Sign Language (ASL) and user-defined gestures.
Main Results:
- The system achieved a maximum F1-score of 98.17% for single-handed gestures and 84.85% for double-handed gestures.
- Recognition of isolated gestures reached a maximum F1-score of 98%, while continuous gestures achieved 83%.
- The system demonstrated high mobility and wireless capability through its implementation on a Raspberry Pi module.
Conclusions:
- The developed gesture-to-speech conversion system offers a lightweight, mobile solution for enhancing communication for the deaf-mute population.
- The system's ability to recognize user-defined gestures with minimal examples (5 per gesture) allows for significant customization.
- Potential applications extend to controlling home appliances and IoT devices, integrating gesture recognition with control interfaces.
