Related Experiment Video
Updated: Aug 6, 2025
Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility
Published on: August 9, 2024
Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery
Pieter De Backer1, Charles Van Praet2, Jente Simoens3
1ORSI Academy, Melle, Belgium; IBiTech-Biommeda, Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, Ghent, Belgium; Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium; Department of Urology, ERN eUROGEN accredited centre, Ghent University Hospital, Ghent, Belgium; Cancer Research Institute Ghent, Ghent University, Ghent, Belgium.
This study introduces a deep learning algorithm for real-time instrument detection in augmented reality (AR) robotic surgery. This enhances safety by improving instrument visibility during procedures like partial nephrectomy and kidney transplantation.
Area of Science:
- Medical Robotics
- Computer-Aided Surgery
- Surgical Navigation
Background:
- Augmented reality (AR) integration in robotic renal surgery faces adoption barriers, including instrument visibility and model alignment issues.
- Current AR systems may superimpose 3D models onto surgical feeds, potentially creating hazardous situations if instruments are not accurately represented.
Observation:
- A novel algorithm utilizing deep learning networks was developed for real-time detection of non-organic items, specifically surgical instruments.
- The algorithm was trained on a large dataset of 65,927 manually labeled instruments across 15,100 frames.
Findings:
- The developed algorithm successfully demonstrated real-time instrument detection during AR-guided robot-assisted partial nephrectomy.
- The system showed generalization capabilities for AR-guided robot-assisted kidney transplantation, proving its versatility.
- The instrument detection setup, running on a standalone laptop, was validated in three hospitals with four surgeons, confirming its feasibility.
Implications:
- Real-time instrument detection significantly enhances the safety and reliability of AR-guided robotic surgery.
- Further optimization of video processing is needed to minimize the current 0.5-second delay for seamless integration.
- Future research should focus on organ deformation tracking and overall AR system optimization for full clinical implementation.
