Related Experiment Video
Updated: Oct 20, 2025
05:12
Using Virtual Reality to Transfer Motor Skill Knowledge from One Hand to Another
Published on: September 18, 2017
547.4K
AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove
Feng Wen1,2,3, Zixuan Zhang1,2,3, Tianyiyi He1,2,3
1Department of Electrical & Computer Engineering, National University of Singapore, Singapore, Singapore.
Nature Communications
|September 11, 2021
Summary
This study introduces an AI-powered system using sensing gloves and deep learning for sign language sentence recognition. It enables bidirectional communication by translating recognized signs into text and audio.
Area of Science:
- Computer Science
- Artificial Intelligence
- Human-Computer Interaction
Background:
- Sign language recognition systems often struggle with sentence-level understanding, limiting communication for the hearing and speech impaired.
- Existing glove-based solutions typically recognize only discrete gestures, failing to meet the demands of daily communication.
Purpose of the Study:
- To develop an artificial intelligence (AI) enabled system for comprehensive sign language recognition and communication.
- To overcome the limitations of discrete gesture recognition by enabling sentence-level understanding.
Main Methods:
- Development of an AI system integrating sensing gloves, a deep learning model, and a virtual reality interface.
- Implementation of both non-segmentation and segmentation-assisted deep learning models for sign language recognition.
- Utilizing a segmentation approach to deconstruct sentences into word units for recognition and reconstruction.
Main Results:
- The system successfully recognized 50 words and 20 sentences.
- The segmentation-assisted deep learning model achieved an average correct recognition rate of 86.67% for new, recombined sentences.
- Sign language recognition results were translated into text and audio within a virtual reality environment.
Conclusions:
- The proposed AI system significantly enhances sign language sentence recognition capabilities.
- The developed system facilitates remote, bidirectional communication between signers and non-signers.
- This technology holds promise for reducing communication barriers for individuals with hearing or speech impairments.
