Related Experiment Video
Updated: Oct 27, 2026
Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation
Published on: September 2, 2025
Camera calibration and tracking for augmented reality during robotic surgery
Lisanne P J Venix1,2, Wouter J Heerink3, Marijn A J Hiep3,4
1Image Guided Surgery Group, Netherlands Cancer Institute, Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands. l.venix@nki.nl.
Purpose:
Robot-assisted surgery (RAS) offers clinical advantages but remains limited by reduced visualization and lack of tactile feedback. Augmented reality (AR), achieved by overlaying preoperative imaging onto laparoscopic video, may mitigate these challenges. This study evaluates the accuracy and clinical feasibility of an AR system that uses electromagnetic (EM) tracking to determine the position of the robotic endoscope and enable real-time image overlay during surgery.
Methods:
For calibration, an EM sensor was rigidly attached to the robotic endoscope and a second sensor to a calibration checkerboard. 35 calibration images were acquired. To determine the required number of calibration images, 2600 random subsets containing 5-30 images were calibrated using RANSAC-based robust estimation and evaluated on 16 separately recorded checkerboard frames using three-dimensional (3D) Euclidean error. Subsequently, a clinically suitable adapter carrying the EM tracker was rigidly mounted on the robotic endoscope and evaluated during 16 robot-assisted pelvic lymph node dissection (RPLND) procedures.
Results:
Calibration accuracy and robustness improved with increasing dataset size. A dataset of 19 images was the smallest dataset size with a median 3D error within 5% of the best observed performance, with a mean 3D Euclidean error of 0.541 ± 0.060 mm. Using the same protocol, the clinical adapter achieved 0.784 ± 0.075 mm. System evaluation during 16 RPLND provided stable, real-time tracking and AR visualization.
Conclusion:
The proposed calibration and tracking approach enables sub-millimeter AR overlay accuracy while allowing continuous tracking of the robotic endoscope through an externally mounted EM tracker. Its successful use during 16 RPLND procedures demonstrates that the method can be integrated into a clinical robotic workflow and supports real-time visualization without substantial procedural modification. These findings provide a technical foundation for further development and validation of image-guided AR applications in robotic-assisted surgery.
