Learning signs with NAO: humanoid robot as a tool for helping to learn Colombian Sign Language

Area of Science:

• Rehabilitation and assistive technologies
• Human-computer interaction
• Computational linguistics

Background:

• Sign languages are crucial for hearing loss rehabilitation, but regional variations and interpreter shortages hinder education.
• Colombian Sign Language (CSL) education faces challenges due to limited interpreter availability.
• Machine learning and computer vision offer potential solutions for sign language translation and education.

Purpose of the Study:

• To evaluate the performance of a novel activity designed to support learning the 11 color-based signs of Colombian Sign Language (CSL).
• To assess the effectiveness of a system utilizing MediaPipe and a Neural Network for CSL color sign recognition.
• To explore the integration of a NAO humanoid robot for enhanced nonverbal interaction in CSL education.

Main Methods:

• Developed an interactive evaluation method with two modes: user-selected and random sign assessment.
• Employed MediaPipe to extract torso and hand coordinates for Neural Network input.
• Evaluated Neural Network performance using webcam and NAO H25 V6 robot camera (wireless streaming).

Main Results:

• The system achieved a 91.6% F1 score with 85.2 ms prediction time using webcam input.
• With the NAO robot's camera, the system achieved a 93.8% F1 score and 2.29 s prediction time.
• Leveraged the NAO robot's 25 degrees of freedom for nonverbal human-robot interactions.

Conclusions:

• The proposed system demonstrates high accuracy and efficiency in recognizing Colombian Sign Language color signs.
• Integration with humanoid robots like NAO offers potential for richer, more interactive educational experiences for the deaf community.
• This work provides a foundation for developing advanced assistive technologies for sign language education in Colombia.