Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance

Area of Science:

• Medical Technology
• Computer Science
• Surgical Robotics

Background:

• Robot-assisted radical prostatectomy (RARP) requires high precision for optimal outcomes.
• Current methods for surgical guidance can be improved with advanced visualization techniques.

Purpose of the Study:

• To develop and evaluate a Deep Learning (DL) and Augmented Reality (AR) system for improved in-vivo organ model alignment during RARP.
• To enhance the precision of a previously published RARP guidance system.

Main Methods:

• A two-step automatic system was implemented using Convolutional Neural Network (CNN) for semantic segmentation of endoscopic images.
• The system aligns a 3D virtual organ model with the 2D endoscopic view, utilizing U-Net for segmentation and evaluating ResNet and MobileNet for processing speed.
• Performance was assessed using a dataset from 5 specialist-tagged endoscopic videos.

Main Results:

• U-Net demonstrated superior segmentation performance.
• MobileNet offered comparable Intersection over Unit (IoU) to ResNet but with nearly double the processing speed.
• The DL-AR system achieved a significantly improved average IoU for the catheter (0.894 vs. 0.339) and reduced anchor point distance (4.160 vs. 12.569).

Conclusions:

• The developed DL-AR system represents a significant advancement in surgical guidance for RARP.
• This approach demonstrates the potential of integrating DL and AR to improve surgical procedure precision and outcomes.
• Future work will focus on further refining the system to address limitations and enhance all aspects of the surgical procedure.