External validation of outcome prediction model for ureteral/renal calculi

We externally validated a previously designed neural network model to predict outcome and duration of passage for ureteral/renal calculi. The model was also evaluated using a 6 mm largest stone dimension cutoff in predicting stone outcome.

The model was previously designed on 301 patients at Albany Medical Center (free shareware from www.uroengineering.com). The model had a prediction accuracy of 86% for passage outcome and 87% for passage duration. In this study we tested the model on a separate 384 patients from 6 different external institutions to assess the prediction accuracy. All patients had a single renal/ureteral calculus by evaluation in an emergency room setting or by primary physicians and were then referred for further treatment. Model accuracy was also compared to using a 6 mm largest stone dimension cutoff in predicting the need for intervention.

Testing on the 384 patients from all 6 external institutions revealed an outcome prediction accuracy of 88%. The area under the ROC curve was 0.9. Using a 6 mm stone size cutoff provided 79% (ROC 0.8) accuracy. The model duration of passage prediction accuracy was 80% (133 patients passed the stone, area under ROC of 0.8).

The model provided high stone outcome prediction accuracy (ROC of 0.9 and 0.8) at the 6 external institutions, comparable to that of the design institution. The model provided higher accuracy than using only the largest stone dimension as a cutoff. Increasing experience will further assess the model's accuracy.