Real-Time Rehabilitation Tracking by Integration of Computer Vision and Virtual Hand.

Area of Science:

• Biomedical Engineering
• Robotics
• Human-Computer Interaction

Background:

• Rehabilitation requires prolonged monitoring, often hindered by the need for in-person visits, posing challenges for remote patients.
• Current telehealth solutions lack the precision needed for accurate rehabilitation tracking.
• There is a significant need for accessible and accurate methods for monitoring rehabilitation progress.

Purpose of the Study:

• To introduce a quantified and accessible framework for analyzing human finger joint torques.
• To enable virtual hand interaction for improved rehabilitation monitoring.
• To validate the accuracy of extracted joint torques for clinical application.

Main Methods:

• Integration of MediaPipe hand landmarks with the MANO hand model in PyBullet for virtual hand representation.
• Utilizing extracted hand positions, rotations, and joint angles as controller inputs for mirroring human movements.
• Designing a PID controller optimized with genetic algorithms for precise joint angle trajectory tracking.

Main Results:

• Successful extraction and analysis of finger joint torques, specifically from metacarpophalangeal and proximal interphalangeal joints.
• Validation of extracted torques with low mean squared errors: $4.346 \times 10^{-4}(~\mathrm{N}. \mathrm{m})^{2}$ and $1.506 \times 10^{-5}(~\mathrm{N}. \mathrm{m})^{2}$, respectively.
• Demonstration of a robust framework for quantified human joint dynamics.

Conclusions:

• The developed framework provides a highly desirable method for tracking rehabilitation progress.
• Quantified joint dynamics information facilitates accurate monitoring using exoskeleton robotics.
• This approach enhances the accessibility and precision of remote rehabilitation care.