Interpolation Methods for MediaPipe Hand Tracking Gaps: A Pose-Based Approach

Kinematics are a relevant measure of movement quality, especially for physically restricted patients. Therefore, consistent landmark detection is necessary. Automated landmarks detection already has existing solutions, but a problem remains: missing frames, where landmarks cannot be automatically detected.

This paper evaluates three interpolation strategies to overcome this issue and thus enable kinematic analysis. These strategies are linear interpolation, the Kalman filter, and the developed pose-based approach. Recordings were obtained using the marker-based VICON system. Participants (n = 3) performed three tasks inspired by a standard stroke assessment. This setup allows the precision of marker-less detection and interpolation to be evaluated.

In comparison with Percentage Root Mean Square Error (PRMSE) and the Procrustes mean disparity, linear interpolation yields the best results. It achieved a mean PRMSE of 36.5 and a mean disparity of 0.266. The pose-based approach, presented here, follows closely with a mean PRMSE of 36.6 and a mean disparity of 0.273. The use of Kalman filtering resulted in a mean disparity of 0.314 and a PRMSE of 35.7. The results were tested positive for significance using the Kolmogorov-Smirnov test.

In subsequent optical evaluations, the pose-based approach demonstrated superior performance; it was possible to observe that the linear approach sometimes lags behind the movement, while the pose-based interpolation follows the movement of the subject. Our results demonstrate that a reliable estimate with consistent landmarks is possible, thus enabling kinematic analysis.