Cloud-Based Personalized sEMG Classification Using Lightweight CNNs for Long-Term Haptic Communication in Deaf-Blind...

Area of Science:

• Assistive Technology
• Human-Computer Interaction
• Biomedical Engineering

Background:

• Deaf-blindness, especially Usher syndrome, creates significant communication and independence barriers.
• Current tactile communication methods often require close proximity and hand-to-hand contact, limiting usability.
• Progressive vision and hearing loss in Usher syndrome necessitate adaptable communication solutions.

Purpose of the Study:

• To develop and validate a novel, cloud-based, AI-assisted gesture recognition and haptic communication system.
• To create a wearable haptic interface that moves tactile interaction from hands to an arm sleeve, preserving dexterity.
• To design a system capable of longitudinal adaptation to users' changing physiological conditions.

Main Methods:

• Utilized surface electromyography (sEMG) to capture muscle activations for gesture recognition.
• Employed personalized, lightweight convolutional neural networks (CNNs) for real-time gesture classification on a central server.
• Developed a wearable haptic interface mounted on an arm sleeve for bidirectional tactile communication.

Main Results:

• Personalized CNN models demonstrated superior accuracy, adaptability, and usability compared to cross-user models.
• The system successfully adapted to users' evolving conditions, including progressive sensory loss.
• Real-time testing with seven participants validated the system's effectiveness and potential for long-term use.

Conclusions:

• The developed platform offers a scalable and longitudinally adaptable solution for non-visual communication.
• This AI-assisted haptic system significantly advances assistive technology for the deaf-blind community.
• The relocation of tactile input/output to an arm sleeve preserves manual dexterity and enables continuous interaction.