Vision-based augmented reality guidance for setting up robot-assisted spine surgery

Robot-assisted procedures improve precision and patient outcomes, but setup often requires tedious manual positioning due to a lack of intuitive guidance. Augmented reality (AR) can assist with coarse alignment by providing visual cues in the correct anatomical context. However, most AR systems require additional setup steps to calibrate the head-mounted displays and imaging systems. We propose an AR guidance approach that uses only RGB video to perform spatial calibration with minimal setup effort.

We establish a shared coordinate frame between the AR headset and the intraoperative imaging system to enable markerless AR guidance. Inter-device poses are estimated directly from RGB video using a Vision Foundation Model (VFM). To improve calibration accuracy in challenging scenarios, such as partial view overlap and long baselines, we apply multi-frame Umeyama optimization with RANSAC, jointly refining scale, rotation, and translation. This process allows users to align the physical robot with the virtual surgical plan with the necessary accuracy.

We evaluate calibration and robot alignment accuracy against an ArUco-based baseline, achieving sub-3 $_{}^{\circ }$ rotation and 15 mm mean translation error without requiring fiducial markers. We further assessed needle placement using a virtual guide that conveyed both the target trajectory and the robot's actuation limits. In all cases, the resulting alignment was within a range that allowed the robot to perform fine adjustments and accurately reach the planned trajectory.

Experimental results show that our vision-based calibration enables accurate needle alignment within the robot's workspace while eliminating setup complexity and fiducial dependence. This demonstrates the feasibility of seamless, markerless AR guidance for robotic procedures via vision-based calibration.