Real-Time Rehabilitation Tracking by Integration of Computer Vision and Virtual Hand

Rehabilitation is a critical component of healthcare services that often requires prolonged monitoring to achieve optimal outcomes. One of the major challenges in rehabilitation tracking is the necessity for patients to visit medical centers, which can be costly and time-consuming, especially for individuals in remote areas. Although telehealth solutions such as video conferencing have been introduced, they are predominantly qualitative and do not provide precise rehabilitation tracking. Consequently, there is a need for a robust, accessible, and accurate method for rehabilitation monitoring. To address this need, we introduce a quantified, easily accessible framework for extracting and analyzing human finger joint torques to enable virtual hand interaction. Hand landmarks from MediaPipe are integrated with the MANO hand model in PyBullet. The hand positions, rotations, and joint angles extracted from the landmarks are used as controller inputs to mirror human hand movements in a virtual environment. A PID controller was designed to accurately track joint angle trajectories, with its parameters optimized using genetic algorithms. The torques extracted from the metacarpophalangeal and proximal interphalangeal joints were validated, achieving mean squared errors of $4.346 \times 10^{-4}(~\mathrm{N}. \mathrm{m})^{2}$ and $1.506 \times 10^{-5}(~\mathrm{N}. \mathrm{m})^{2}$, respectively. This quantification and accessibility of human joint dynamics information is highly desirable for tracking rehabilitation progress using exoskeleton robotics.