Deep learning-based eye sign communication system for people with speech impairments.

Area of Science:

• Computer Science
• Biomedical Engineering
• Human-Computer Interaction

Background:

• Individuals with motor and speech impairments face communication barriers.
• Augmentative and Alternative Communication (AAC) methods include eye-gaze and eye signs.
• Current eye-gaze systems require calibration and are sensitive to lighting; eye sign research is nascent.

Purpose of the Study:

• To develop an eye sign-based communication system for individuals with speech impairments.
• To overcome limitations of existing eye-gaze technology, such as calibration and lighting sensitivity.
• To advance research in eye sign communication.

Main Methods:

• A deep learning-based system using a standard webcam to capture eye sign patterns.
• Conversion of eye signs into text (alphabets, words, sentences) with visual display.
• Inclusion of vocal prompts, text prediction, and calibration-free operation in varied lighting.

Main Results:

• High accuracy rates achieved: 98% for alphabet, 99% for word, and 99% for sentence formation.
• Demonstrated robustness across different lighting conditions without calibration.
• Positive impact experiments with participants aged 18-35 years.

Conclusions:

• The proposed eye sign communication system is robust and accurate.
• It offers a significant advancement for individuals with speech impairments.
• The system has the potential to greatly improve communication accessibility.