ASSIST-U: A system for segmentation and image style transfer for ureteroscopy

Area of Science:

• Medical Imaging
• Surgical Simulation
• Artificial Intelligence

Background:

• Ureteroscopy is crucial for kidney stones and upper tract urothelial carcinoma but poses training challenges.
• Current endoscopic training lacks patient-specific simulators and standardized visualization tools.
• Surgical errors, like missing stones or tumors, necessitate repeat operations.

Purpose of the Study:

• To introduce the ASSIST-U system for creating realistic ureteroscopy images and videos.
• To address the unmet need for patient-specific training simulators in ureteroscopy.
• To enhance surgical visualization during endoscopic procedures.

Main Methods:

• Utilized preoperative computerized tomography (CT) images as input.
• Employed a 3D UNet model for automatic CT image segmentation and 3D surface construction.
• Applied skeletonization for surface rendering and Contrastive Unpaired Translation (CUT) for image synthesis.
• Trained a style transfer model to generate realistic ureteroscopy visuals.

Main Results:

• The CT segmentation model achieved a Dice score of 0.853 ± 0.084 in cross-validation.
• Synthesized ureteroscopy images showed significant visual plausibility.
• Kernel inception distance improved from 0.198 (rendered) to 0.089 (synthesized) compared to real images.
• The complete pipeline from CT to synthesized images was qualitatively validated.

Conclusions:

• The ASSIST-U system demonstrates potential for improving surgical visualization in kidney ureteroscopy.
• The technology offers a promising approach for developing patient-specific training simulators.
• This AI-driven system can aid surgeons by providing realistic visual feedback during procedures.