A multi-view pipeline and benchmark dataset for 3D hand pose estimation in surgery

Area of Science:

• Computer Vision
• Medical Robotics
• Surgical Technology

Background:

• Accurate 3D hand pose estimation is crucial for surgical applications like skill assessment and robot-assisted interventions.
• Surgical environments present significant challenges, including difficult lighting, occlusions, and uniform hand appearance due to gloves.
• A scarcity of annotated datasets hinders reliable model training for surgical hand pose estimation.

Purpose of the Study:

• To develop a robust multi-view pipeline for 3D hand pose estimation in surgical settings.
• To create a benchmark dataset for evaluating hand pose estimation models in operating room environments.
• To provide a framework for surgical motion analysis and objective skill assessment.

Main Methods:

• A multi-view pipeline combining person detection, whole-body pose estimation, and 2D hand keypoint prediction.
• Utilized off-the-shelf pretrained models without domain-specific fine-tuning.
• Introduced a new benchmark dataset with over 68,000 frames and 3,000 annotated 2D/3D hand poses from a physical operating room replica.

Main Results:

• The proposed pipeline significantly outperforms state-of-the-art methods in both 2D and 3D hand pose estimation.
• Achieved up to a 31% reduction in 2D joint error and a 76% reduction in 3D joint position error.
• Demonstrated robustness across increasing scene complexity, including motion and partial occlusions.

Conclusions:

• The multi-view pipeline effectively estimates 3D hand poses in surgical environments without retraining.
• The developed dataset serves as a valuable resource for advancing surgical motion analysis and skill assessment research.
• This work establishes a baseline framework for future developments in surgical hand tracking and analysis.