Machine Learning-Assisted Gesture Sensor Made with Graphene/Carbon Nanotubes for Sign Language Recognition

Area of Science:

• Materials Science
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Gesture sensors are crucial for human-computer interfaces but struggle with simultaneous high sensitivity and wide response range.
• Existing sensors face limitations in accurately capturing nuanced human movements for complex interactions.

Purpose of the Study:

• To develop a novel gesture sensor inspired by natural spider silk structures.
• To enhance the sensitivity and response range of gesture sensing technology.
• To integrate the sensor with deep learning for advanced gesture recognition.

Main Methods:

• Fabrication of a core-shell structured gesture sensor mimicking spider silk.
• Characterization of sensor performance, including gauge factor and response range.
• Development of a gesture recognition system using deep learning, sliding window technology, and large language models.

Main Results:

• The proposed sensor exhibits a high gauge factor (up to 340) and a wide response range (60%).
• The integrated system achieved 99% accuracy in single gesture recognition.
• Continuous gesture (sentence) recognition reached 97% accuracy using sliding window and large language model techniques.

Conclusions:

• The novel core-shell gesture sensor significantly enhances sensitivity and response range.
• The combination of advanced sensor technology and deep learning offers a high-performance solution for precise gesture recognition.
• This technology holds potential for improving daily communication for sign language users and advancing human-computer interfaces.