Ambient intelligence framework for real-time speech-to-sign translation

Area of Science:

• Assistive Technology
• Human-Computer Interaction
• Sign Language Recognition

Background:

• Sign language is crucial for communication among the deaf and hard-of-hearing (Deaf/HH) community.
• Video streaming enables global communication but often lacks sign language support.
• Existing solutions may not offer real-time translation for dynamic content.

Purpose of the Study:

• To develop a prototype assistive device for real-time speech-to-sign translation.
• To enable Deaf/HH individuals to access and understand mobile streaming video content.
• To improve translation accuracy using eye-tracking based user-satisfaction detection and dynamic learning.

Main Methods:

• Development of a prototype speech-to-sign translation system.
• Implementation of pipelined and parallel processing for real-time translation.
• Integration of eye-tracking for user-satisfaction detection to dynamically refine translations.
• Conducting performance and usability evaluations with nine deaf participants.

Main Results:

• Real-time translation with viewer's attention feedback reduced sign error rates (SER) by 16%.
• Translation accuracy, measured by the Bilingual Evaluation Understudy (BLEU) metric, increased by 5.4% compared to a baseline system.
• Usability studies confirmed the device is pleasant and satisfying for deaf users.

Conclusions:

• The developed assistive device enhances real-time speech-to-sign translation accuracy and reduces errors.
• Eye-tracking feedback integration proves effective in improving translation quality for video content.
• The device shows potential for increasing engagement of deaf individuals in daily activities and digital content consumption.