Enhancing open-surgery gesture recognition using 3D pose estimation

Purpose Surgical gestures are fundamental components of surgical procedures, encompassing actions such as cutting, suturing, and knot-tying. Gesture recognition plays a pivotal role in the automatic analysis of surgical data. Although recent advancements have improved surgical gesture recognition, much of the existing research relies on simulations or minimally invasive surgery data, failing to capture the complexities of open surgery. In this study, we introduce and employ a new open surgery dataset focused on closing incisions after saphenous vein harvesting. Methods Our goal is to improve gesture recognition accuracy by incorporating tool pose estimation and 3D hand pose predictions of surgeons. We employ MS-TCN++  and LTContext  for gesture recognition, and further enhance performance through an ensemble of models using different modalities-video, tool pose, and hand pose data.Results The results reveal that using an ensemble model combining all three modalities provides a substantial improvement over video-only approaches, leading to statistically significant gains across multiple evaluation metrics. We further demonstrate that the model can rely solely on hand and tool poses, completely discarding the video input, while still achieving comparable performance. Additionally, we show that an ensemble model using only hand and tool poses produces results that are either: statistically significantly better than using video alone, or not statistically significantly different.Conclusion This study highlights the effectiveness of integrating multimodal data for surgical gesture recognition. By combining video, hand pose, and tool pose information, our approach achieves higher accuracy and robustness compared to video-only methods. Moreover, the comparable performance of pose-only models suggests a promising, privacy-preserving alternative for surgical data analysis.