Towards a zero-shot low-latency navigation for open surgery augmented reality applications.

Area of Science:

• Medical Imaging
• Computer-Assisted Surgery
• Augmented Reality

Background:

• Augmented reality (AR) with head-mounted displays (HMDs) enhances surgical navigation by overlaying 3D models onto patient anatomy.
• Open liver surgery can significantly benefit from AR navigation for tumor and critical structure identification.
• Current AR methods lack automatic, markerless, and robust solutions for real-world surgical challenges like occlusion and organ motion.

Purpose of the Study:

• To introduce a novel multi-device approach for automatic, markerless live navigation in open liver surgery.
• To enhance AR visualization and interaction using a HoloLens 2 HMD with precise registration.
• To overcome limitations of existing methods by addressing occlusion and organ motion.

Main Methods:

• A multi-device system combining HoloLens 2 and an Intel RealSense RGB-D camera for precise registration.
• Intraoperative RGB-D segmentation and preoperative CT data for virtual liver model registration.
• AR-prompted Segment Anything Model (SAM) for robust, in-situ liver segmentation without retraining.
• Double Exponential Smoothing (DES) for forecasting registration results to mitigate algorithmic latency.

Main Results:

• Phantom study demonstrated comparable registration errors (8.31–18.78 mm TRE) to prior work.
• High success rates achieved even with significant liver motion and high occlusion factors.
• Algorithmic latency of 79.8 ms per frame was bypassed using forecasting, with median errors below 2 mm and 1.5 degrees.

Conclusions:

• This work pioneers markerless in-situ AR visualization by integrating multi-device registration, forecasting, and a foundation model.
• The approach offers reliable, precise AR registration of surgical targets with low latency.
• The methodology is adaptable to other surgical applications and AR hardware.