The Evaluation of Computer Vision-Based Automated Performance Metrics for Endoscopic Kidney Stone Surgery

Background: The assessment of surgical competency is essential for clinical training and safety. No objective, real-time tools exist to evaluate competency during endoscopic stone operation. We sought to apply endoscopic computer vision models to define automated performance metrics (APMs) from videos of flexible ureteroscopy. Materials and Methods: We assessed three APMs for endoscopic treatment of kidney stones, including percentage of frames without stone visibility, screen occupancy by stone, and frame-to-frame change in stone occupancy. Surgical videos of a surgeon performing either stone localization or stone ablation were recorded. Using our previously validated computer vision model for endoscopic stone segmentation, APMs were compared between experts (fellowship-trained endourologists) and trainees. Results: Forty-six videos, including 28 of stone localization and 18 of stone laser ablation, were analyzed from nine surgeons (three experts and six trainees). During stone localization, trainee videos had a higher percentage of frames without visible stone (4% vs 27%, p < 0.01) and lower screen occupancy by stone (5% vs 14%, p = 0.03) compared with expert videos. During laser ablation, trainee videos had a higher frame-to-frame change in stone occupancy (3% vs 2%, p < 0.01) compared with expert videos. Conclusions: APMs from computer vision methods differ between expert and trainee surgical videos of endoscopic kidney stone treatment. These metrics could be used to objectively assess skill evaluation and acquisition.