A multi-view pipeline and benchmark dataset for 3D hand pose estimation in surgery

Accurate 3D hand pose estimation supports surgical applications such as skill assessment and robot-assisted interventions. However, surgical environments pose severe challenges, including intense and localized lighting, frequent occlusions by instruments or staff, and uniform hand appearance due to gloves, combined with a scarcity of annotated datasets for reliable model training.

We propose a multi-view pipeline for 3D hand pose estimation in surgical contexts that requires no domain-specific fine-tuning and relies on off-the-shelf pretrained models. The pipeline combines person detection, whole-body pose estimation, and 2D hand keypoint prediction on tracked hand crops, followed by constrained 3D optimization. We also introduce a benchmark dataset with over 68,000 frames and 3,000 manually annotated 2D hand poses with triangulated 3D ground truth, recorded in a high-fidelity physical operating room replica.

Quantitative evaluation on the proposed dataset shows that the pipeline consistently outperforms state-of-the-art baselines for both two- and three-dimensional hand pose estimation. The method substantially improves joint localization accuracy, achieving up to a 31% reduction in 2D joint error and a 76% reduction in 3D joint position error. The approach remains robust across increasing levels of scene complexity, including motion and partial occlusions.

The proposed multi-view pipeline demonstrates the potential of combining robust detection, tracking, and geometric optimization for three-dimensional hand pose estimation in surgical environments without domain-specific retraining. Together with the introduced dataset, this work provides a baseline framework and benchmarking resource to support future research on surgical motion analysis and objective skill assessment.