Towards markerless surgical tool and hand pose estimation

Area of Science:

• Computer-assisted surgery
• Medical imaging
• Robotics

Background:

• Tracking surgical tools and activity is crucial for computer-assisted surgery.
• Markerless hand and instrument pose estimation is essential for realistic surgical scenarios.

Purpose of the Study:

• To present a data generation framework, dataset, and baseline methods for markerless hand and instrument pose estimation.
• To facilitate further research in realistic surgical scenarios.

Main Methods:

• Developed a rendering pipeline for generating inexpensive, realistic synthetic data for model pretraining.
• Proposed a pipeline for capturing and labeling real data with ground truth hand and object poses.
• Presented three state-of-the-art RGB-based pose estimation baselines.

Main Results:

• Evaluated three baseline models on the generated synthetic and real datasets.
• The best baseline achieved an average tool 3D vertex error of 16.7 mm on synthetic data and 13.8 mm on real data.
• Performance is comparable to state-of-the-art RGB-based hand/object pose estimation.

Conclusions:

• The study proposes the first synthetic and real data generation pipelines for hand and object pose labels in open surgery.
• Presented baseline models for RGB-based object and hand pose estimation.
• The synthetic data generation pipeline can help overcome data limitations in surgery and be applied to other medical fields.