Related Experiment Video
Updated: Sep 12, 2025
10:23
Technical Approach for Infrared Tracking for Soft Tissue Navigation with a Holographic Head-Mounted Display and Preclinical Validation
Published on: September 2, 2025
27
Towards a zero-shot low-latency navigation for open surgery augmented reality applications
Michael Schwimmbeck1,2, Serouj Khajarian3,4, Christopher Auer3
1Research Group Medical Technologies, University of Applied Sciences Landshut, Landshut, Germany. michael.schwimmbeck@haw-landshut.de.
Summary
This study presents a novel augmented reality (AR) system for liver surgery, improving navigation accuracy and reducing latency using markerless tracking and AI-driven segmentation. The system demonstrates robust performance even with organ motion and occlusion.
Area of Science:
- Medical Imaging
- Computer-Assisted Surgery
- Augmented Reality
Background:
- Augmented reality (AR) with head-mounted displays (HMDs) enhances surgical navigation by overlaying 3D models onto patient anatomy.
- Open liver surgery can significantly benefit from AR navigation for tumor and critical structure identification.
- Current AR methods lack automatic, markerless, and robust solutions for real-world surgical challenges like occlusion and organ motion.
Purpose of the Study:
- To introduce a novel multi-device approach for automatic, markerless live navigation in open liver surgery.
- To enhance AR visualization and interaction using a HoloLens 2 HMD with precise registration.
- To overcome limitations of existing methods by addressing occlusion and organ motion.
Main Methods:
- A multi-device system combining HoloLens 2 and an Intel RealSense RGB-D camera for precise registration.
- Intraoperative RGB-D segmentation and preoperative CT data for virtual liver model registration.
- AR-prompted Segment Anything Model (SAM) for robust, in-situ liver segmentation without retraining.
- Double Exponential Smoothing (DES) for forecasting registration results to mitigate algorithmic latency.
Main Results:
- Phantom study demonstrated comparable registration errors (8.31–18.78 mm TRE) to prior work.
- High success rates achieved even with significant liver motion and high occlusion factors.
- Algorithmic latency of 79.8 ms per frame was bypassed using forecasting, with median errors below 2 mm and 1.5 degrees.
Conclusions:
- This work pioneers markerless in-situ AR visualization by integrating multi-device registration, forecasting, and a foundation model.
- The approach offers reliable, precise AR registration of surgical targets with low latency.
- The methodology is adaptable to other surgical applications and AR hardware.
