An augmented reality overlay for navigated prostatectomy using fiducial-free 2D-3D registration

Area of Science:

• Medical Robotics
• Surgical Navigation
• Computer-Assisted Surgery

Background:

• Markerless navigation in minimally invasive surgery remains a significant challenge.
• Clinical practice often utilizes monocular endoscopes, unlike many navigation systems requiring stereoscopic images.
• A lack of automatic video-based navigation systems for prostatectomies exists.

Purpose of the Study:

• To develop methods for markerless navigation in robot-assisted prostatectomies, addressing the limitations of monocular endoscopes and the absence of automatic video-based systems.
• To create a semi-automatic augmented reality overlay for navigated prostatectomy.
• To enable camera pose estimation relative to the prostate using visual cues.

Main Methods:

• Developed a method using visual cues drawn on the prostate for initial manual alignment, creating matching 2D and 3D landmarks.
• Tracked visual cues in endoscopic video from a key frame.
• Explored both PnP-RANSAC and differentiable rendering for 2D-3D registration in each frame.

Main Results:

• Experiments were conducted on both synthetic and in vivo data.
• Differentiable rendering achieved a median target registration error of 6.11 mm on synthetic data.
• Both PnP-RANSAC and differentiable rendering were found to be feasible methods for 2D-3D registration.

Conclusions:

• Demonstrated a video-based markerless augmented reality overlay for navigated prostatectomy.
• Utilized visual cues as anchors for the augmented reality system.
• The developed method shows promise for improving navigation in minimally invasive prostatectomies.