Related Experiment Video
Updated: Nov 3, 2025
Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation
Published on: September 2, 2025
Preclinical evaluation of a markerless, real-time, augmented reality guidance system for robot-assisted radical
Megha Kalia1, Apeksha Avinash2, Nassir Navab3
1Electrical and Computer Engineering, University of British Columbia, 2329 West Mall, Vancouver, BC, V6T 1Z4, Canada. mkalia@ece.ubc.ca.
Purpose:
Intra-operative augmented reality (AR) during surgery can mitigate incomplete cancer removal by overlaying the anatomical boundaries extracted from medical imaging data onto the camera image. In this paper, we present the first such completely markerless AR guidance system for robot-assisted laparoscopic radical prostatectomy (RALRP) that transforms medical data from transrectal ultrasound (TRUS) to endoscope camera image. Moreover, we reduce the total number of transformations by combining the hand-eye and camera calibrations in a single step.
Methods:
Our proposed solution requires two transformations: TRUS to robot, [Formula: see text], and camera projection matrix, [Formula: see text] (i.e., the transformation from endoscope to camera image frame). [Formula: see text] is estimated by the method proposed in Mohareri et al. (in J Urol 193(1):302-312, 2015). [Formula: see text] is estimated by selecting corresponding 3D-2D data points in the endoscope and the image coordinate frame, respectively, by using a CAD model of the surgical instrument and a preoperative camera intrinsic matrix with an assumption of a projective camera. The parameters are estimated using Levenberg-Marquardt algorithm. Overall mean re-projection errors (MRE) are reported using simulated and real data using a water bath. We show that [Formula: see text] can be re-estimated if the focus is changed during surgery.
Results:
Using simulated data, we received an overall MRE in the range of 11.69-13.32 pixels for monoscopic and stereo left and right cameras. For the water bath experiment, the overall MRE is in the range of 26.04-30.59 pixels for monoscopic and stereo cameras. The overall system error from TRUS to camera world frame is 4.05 mm. Details of the procedure are given in supplementary material.
Conclusion:
We demonstrate a markerless AR guidance system for RALRP that does not need calibration markers and thus has the capability to re-estimate the camera projection matrix if it changes during surgery, e.g., due to a focus change.
