An assistive interface protocol for communication between visually and hearing-speech impaired persons in internet...

Area of Science:

• Assistive Technology
• Human-Computer Interaction
• Rehabilitation Engineering

Background:

• Effective communication is crucial for social integration and quality of life.
• Individuals with hearing-speech and visual impairments face significant communication barriers.
• Existing assistive technologies may not fully address the complex needs of dual-impairment scenarios.

Purpose of the Study:

• To design and develop a generic assistive system for independent conversations.
• To create a communication platform for hearing-speech impaired and visually impaired persons.
• To enable seamless interaction between individuals with different sensory and speech impairments.

Main Methods:

• Developed a software system using Python and HTML.
• Implemented speech-to-text/gesture and text/gesture-to-speech conversion functionalities.
• Utilized Convolutional Neural Network (CNN) deep learning and MediaPipe for real-time hand-gesture recognition and speech generation.

Main Results:

• Achieved average prediction times under 1s for audio-words and 2s for hand-gestures.
• Demonstrated high accuracy (0.9996) and low loss (0.0008) in CNN-based hand-gesture recognition.
• Validated satisfactory performance in recognizing alphabet-specific hand-gestures via confusion matrix analysis.

Conclusions:

• The developed software prototype facilitates exclusive internet-based communication between visually impaired and hearing-speech impaired individuals.
• The system's interface supports communication among visually impaired persons only or hearing-speech impaired persons only.
• Future work can extend the design to include multi-modal impairments for a universal assistive communication device.