Markerless Tracking of Robotic Surgical Instruments With Head Mounted Display for Augmented Reality Applications

In robotic-assisted laparoscopic surgery, an assistant surgeon stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the assistant include navigating new instruments into the surgeon's field-of-view and passing in or retracting materials from the body using hand-held tools. We previously developed ARssist, an augmented reality application based on an optical see-through head-mounted display (HMD), to aid the assistant. Localization of the HMD with respect to the robot was achieved via the attachment of markers. In this paper, we propose a novel markerless tracking method for robotic instruments using a HoloLens 2 HMD. We first run off-the-shelf YOLOv11 and SAMURAI (an adaptation of Segment Anything 2) networks to detect instrument primitives (shaft lines and keypoints). We then recover full 6D poses via a geometrically interpretable pipeline combining perspective-n-point (PnP) and a multi-view least-squares optimization. We experimentally compare the markerless tracking accuracy to a baseline marker-based tracking solution, and show similar instrument tip accuracy. This suggests that the markerless method is an acceptable substitute to marker-based tracking for this augmented reality application, while avoiding workflow issues with sterilizing and attaching markers.