A graphical user interface for editing keypoints from human pose estimation algorithms

Human pose estimation algorithms offer exciting potential for performing markerless kinematic tracking of human movement using only simple videos recorded from common household devices (e.g., smartphones, tablets, webcams). However, these algorithms can be limited by mis-tracking or failure to track certain "keypoints" (i.e., digital markers) associated with specific anatomical locations. These tracking errors can be caused by a wide variety of factors, including hallucinations, occlusions, and noise (among others). There is a need for new tools capable of editing the keypoints resulting from applications of human pose estimation algorithms to ensure accurate kinematic tracking throughout the video of interest. Here, we developed and tested a new pose editor tool that provides a graphical user interface (GUI) enabling users to 1) correct mis-tracked or missing keypoints by manually relocating them to their optimal visual locations directly on the video and 2) run pose estimation directly within the GUI for improved accessibility. We tested the tool on a previous dataset of pose estimation-based keypoints of persons with stroke walking on a treadmill. We observed significant improvements in the mean absolute error between keypoint-based kinematics and ground-truth motion capture data and significant improvements in the correlations between the time series data of the keypoint-based kinematics and the ground-truth motion capture data. Finally, we also observed that the time spent performing manual editing showed a significant positive relationship with the degree of improvement in the kinematic estimates. We have made all software freely available at our GitHub repository: https://github.com/JeffZC/pose-editor.