Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware

BackgroundAugmented Reality (AR) during Robotic-Assisted Radical Prostatectomy (RARP) provides surgical guidance by overlaying patient-specific 3D Virtual Models (VM) onto the intraoperative view. A main drawback is instrument occlusion, where VM overlaps the surgical tools, disturbing the augmented view. Deep learning (DL)-based desocclusion addresses this issue, but current implementation rely on expensive, specialized hardware.MethodsWe developed an AR desocclusion pipeline running on consumer-grade hardware (∼1150 US Dollars). For model selection, we trained and benchmarked 21 encoder architectures paired with a U-Net decoder. The selected model was integrated into a macOS application using CoreML inference on the Apple Neural Engine. Memory usage, thermal state and framerate were monitored on a continuous 81-minute console session. Photon-to-photon latency (from surgical field to the surgeon eye) was measured on the DaVinci® Xi using a microcontroller-phototransistor setup.ResultsFastViT-S12/Unet offered the highest accuracy among models meeting our inference-time budget (∼2 ms inference), achieving a Dice score of 95.1% (95% CI, 94.3-95.7) internally, 91.6% (95% CI, 90.2-92.9) and 90.2% (95% CI, 84.7-94.4) on externals tests, with a mean inference time of 1.89 ± 0.36 ms. The pipeline maintained 57.98 frames per second using 1.47 GB of unified memory with a nominal thermal state. On the surgical console, photon-to-photon median latency increased from 60 ms (Q1-Q3: 57.5-62.5) to 120.5 ms (Q1-Q3: 111-131) with the AR pipeline.ConclusionsReal-time AR-RARP with DL-based desocclusion is achievable on low-cost hardware with an added latency remaining within ranges reported as technically acceptable for intraoperative use.