Development and validation of a surgical tool recognition and localization strategy in robotic surgeries

Despite the rapid progress of robotic surgery, intelligent object recognition and autonomous computer-assisted interventions continues to face limitations. Algorithms designed for the identification and localization of surgical tool tips are pivotal in augmenting the interactive collaboration between the robotic surgical system and its operator. In the present study, our aim to formulate a strategy proficient in recognizing and pinpointing commonly used surgical tools in robotic surgery.

A strategy for object detection was conceptualized and validated across two datasets. The first dataset provided by the Medical Image Computing and Computer Assisted Intervention Society (MICCAI) consisted of 24,695 video clips. The second dataset comprised surgical videos from 35 clinical cases in our clinical center, encompassing a total of 392 video clips with 53,128 images. Our approach employs CSPNeXt as the backbone network, incorporates OSTrack for pseudo-labeling and Kalman filtering for data fusion. Our model is openly accessible on GitHub.

In the MICCAI test set, our strategy achieved a mean Average Precision of 0.4669. In the second video test set, the recall rate (RR) and precision rate (PR) for seven types of tools ranged from 79.36% to 99.75% and from 57.65% to 97.35%, respectively. RR and PR of bipolar forceps, needle driver and cautery hook all exceeded 94%.

An effective strategy for the recognition of surgical tools and tips was successfully developed and validated. This strategy shows promise in actively supporting surgical procedures and has the potential to alleviate the workload on human operators during future algorithm training.