Related Experiment Video
Updated: May 18, 2026
09:41
A Pipeline for 3D Multimodality Image Integration and Computer-assisted Planning in Epilepsy Surgery
Published on: May 20, 2016
Generative AI pipeline with model-guided filtering for sim-to-real transfer in surgical imaging
Pietro Leoncini1, Francesco Marzola2, Matteo Pescio1
1Department of Surgical Sciences, Università degli Studi di Torino, Corso Dogliotti 14, Turin, TO 10126, Italy; DIMEAS, Italy.
Summary
Generating realistic surgical data for computer vision is challenging. This study introduces a pipeline using synthetic data, generative enhancement, and filtering to significantly improve robotic surgery simulation accuracy without real-world annotations.
Area of Science:
- Computer Vision
- Robotic Surgery
- Data Augmentation
Background:
- Automating surgical suturing necessitates robust computer vision systems.
- Acquiring annotated real surgical datasets is a significant bottleneck due to cost and difficulty.
Purpose of the Study:
- To develop a data-centric pipeline for enhancing sim-to-real transfer in surgical vision.
- To improve the performance of computer vision models for surgical automation without relying on real annotated data.
Main Methods:
- Utilized Unity for synthetic data generation with type-based and part-based instrument annotations.
- Employed CycleGAN-TURBO and Real-ESRGAN for generative realism boosting and high-resolution restoration.
- Implemented a YOLO-based selector model for automated filtering of enhanced synthetic data based on Dice similarity scoring.
Main Results:
- The hybrid curated dataset, combining filtered enhanced synthetic data, achieved a Dice score of 0.44 in the part-based configuration and 0.65 in the type-based configuration.
- This represents a substantial improvement over baseline models trained solely on synthetic data (0.17 part-based, 0.384 type-based).
- Data quality improvements, driven by generative enhancement and filtering, were more impactful than fine-tuning strategies.
Conclusions:
- Automated curation of generative outputs is critical for successful sim-to-real transfer in surgical vision.
- The proposed pipeline enables scalable, low-cost dataset creation for developing reliable surgical perception systems.
- This work provides a reproducible foundation for advancing autonomy in surgical robotics.