Marker Data Enhancement for Markerless Motion Capture

Human pose estimation models can measure movement from videos at a large scale and low cost; however, open-source pose estimation models typically detect only sparse keypoints, which leads to inaccurate joint kinematics. OpenCap, a freely available service for researchers to measure movement from videos, mitigates this issue using a deep learning model-the marker enhancer-that transforms sparse video keypoints into dense anatomical markers. However, OpenCap performs poorly on movements not included in the training data. Here, we create a much larger and more diverse training dataset and develop a more accurate and generalizable marker enhancer.

We compiled marker-based motion capture data from 1176 subjects and synthesized 1433 hours of video keypoints and anatomical markers to train the marker enhancer. We evaluated its accuracy in computing kinematics using both benchmark movement videos and synthetic data representing unseen, diverse movements.

The marker enhancer improved kinematic accuracy on benchmark movements (mean error: 4.1$^\circ$, max: 8.7$^\circ$) compared to using video keypoints (mean: 9.6$^\circ$, max: 43.1$^\circ$) and OpenCap's original enhancer (mean: 5.3$^\circ$, max: 11.5$^\circ$). It also better generalized to unseen, diverse movements (mean: 4.1$^\circ$, max: 6.7$^\circ$) than OpenCap's original enhancer (mean: 40.4$^\circ$, max: 252.0$^\circ$).

Our marker enhancer demonstrates both improved accuracy and generalizability across diverse movements.

We integrated the marker enhancer into OpenCap, thereby offering its thousands of users more accurate measurements across a broader range of movements.