Beyond the joystick: deep learning games for hand movement recovery

Area of Science:

• Computer Vision
• Rehabilitation Technology
• Artificial Intelligence

Background:

• Traditional hand rehabilitation can be repetitive and lack engagement.
• There is a need for accessible, low-cost, and interactive therapeutic tools.
• Gamification and AI offer potential to improve patient motivation and outcomes.

Purpose of the Study:

• To explore Deep Learning (DL) for real-time hand and gesture recognition in hand rehabilitation.
• To develop a gamified system for enhancing cognitive function and hand-eye coordination.
• To create an accessible and engaging platform for dexterity recovery.

Main Methods:

• Utilized pre-trained Convolutional Neural Network (CNN) models with Google's MediaPipe Library for hand recognition.
• Redeveloped classic arcade games (Pong, Tetris, Fruit Ninja) and a Virtual Keyboard into gesture-controlled rehabilitation tools.
• Implemented a web-based interface using Phaser.js, a scoring system, and evaluated usability with the System Usability Scale (SUS).

Main Results:

• Demonstrated consistent gesture recognition accuracy and stable game control with 15 participants.
• Achieved favorable user responses in usability evaluations, with SUS scores exceeding the industry benchmark.
• Indicated that participants found the system intuitive, engaging, and effective for rehabilitation.

Conclusions:

• Confirmed the feasibility of monocular camera-based computer vision for hand rehabilitation.
• The proposed system offers an accessible, interactive, and affordable alternative to existing tools.
• Gesture recognition integrated with gamified interfaces effectively supports dexterity recovery and user motivation, laying groundwork for future AI rehabilitation platforms.