Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos

Objective.To assess the performance and added value of processing complete digital endoscopic video sequences for the automatic recognition of stone morphological features during a standard-of-care intra-operative session.Approach.A computer-aided video classifier was developed to predictin-situthe morphology of stone using an intra-operative digital endoscopic video acquired in a clinical setting. Using dedicated artificial intelligence (AI) networks, the proposed pipeline selects adequate frames in steady sequences of the video, ensures the presence of (potentially fragmented) stones and predicts the stone morphologies on a frame-by-frame basis. The automatic endoscopic stone recognition (A-ESR) is subsequently carried out by mixing all collected morphological observations.Main results.The proposed technique was evaluated on pure (i.e. include one morphology) and mixed (i.e. include at least two morphologies) stones involving 'Ia/Calcium Oxalate Monohydrate' (COM), 'IIb/Calcium Oxalate Dihydrate' (COD) and 'IIIb/Uric Acid' (UA) morphologies. The gold standard ESR was provided by a trained endo-urologist and confirmed by microscopy and infra-red spectroscopy. For the AI-training, 585 static images were collected (349 and 236 observations of stone surface and section, respectively) and used. Using the proposed video classifier, 71 digital endoscopic videos were analyzed: 50 exhibited only one morphological type and 21 displayed two. Taken together, both pure and mixed stone types yielded a mean diagnostic performances as follows: balanced accuracy = [88 ± 6] (min = 81)%, sensitivity = [80 ± 13] (min = 69)%, specificity = [95 ± 2] (min = 92)%, precision = [78 ± 12] (min = 62)% and F1-score = [78 ± 7] (min = 69)%.Significance.These results demonstrate that AI applied on digital endoscopic video sequences is a promising tool for collecting morphological information during the time-course of the stone fragmentation process without resorting to any human intervention for stone delineation or the selection of adequate steady frames.