Robust recovery of human motion from video using Kalman filters and virtual humans

In sport science, as in clinical gait analysis, optoelectronic motion capture systems based on passive markers are widely used to recover human movement. By processing the corresponding image points, as recorded by multiple cameras, the human kinematics is resolved through multistage processing involving spatial reconstruction, trajectory tracking, joint angle determination, and derivative computation. Key problems with this approach are that marker data can be indistinct, occluded or missing from certain cameras, that phantom markers may be present, and that both 3D reconstruction and tracking may fail. In this paper, we present a novel technique, based on state space filters, that directly estimates the kinematical variables of a virtual mannequin (biomechanical model) from 2D measurements, that is, without requiring 3D reconstruction and tracking. Using Kalman filters, the configuration of the model in terms of joint angles, first and second order derivatives is automatically updated in order to minimize the distances, as measured on TV-cameras, between the 2D measured markers placed on the subject and the corresponding back-projected virtual markers located on the model. The Jacobian and Hessian matrices of the nonlinear observation function are computed through a multidimensional extension of Stirling's interpolation formula. Extensive experiments on simulated and real data confirmed the reliability of the developed system that is robust against false matching and severe marker occlusions. In addition, we show how the proposed technique can be extended to account for skin artifacts and model inaccuracy.