External Validation Demonstrates Machine Learning Models Outperform Human Experts in Prediction of Objective and
Glenn T Werneburg1, Eric A Werneburg2, Howard B Goldman1
1Department of Urology, Glickman Urological and Kidney Institute, Cleveland Clinic Foundation, Cleveland, OH.
Objective:
To predict treatment response for overactive bladder (OAB) for a specific patient remains elusive. We sought to develop accurate models using machine learning for prediction of objective and patient-reported treatment response to intravesical botulinum toxin (OBTX-A) injection. We sought to validate the models in a challenging setting using an external dataset of a markedly different patient cohort and dosing regimen. We hypothesized the model would outperform human experts and top available algorithms.
Methods:
Algorithms using "operator splitting" designed for accuracy and efficiency even in small training datasets with variable completeness, were trained to predict objective response and patient-reported symptomatic improvement using the ROSETTA trial cohort and validated using the ABC trial cohort of patients who underwent OBTX-A. Areas under the curve (AUC) of algorithms were compared to the top publicly-available machine-learning classifier XGBoost, logistic regression with cross validation, and human expert predictions in the external validation set.
Results:
In the validation set, the operator splitting neural network had AUC of 0.66 and outperformed XGBoost with DART (top available machine-learning classifier, AUC: 0.58), logistic regression (AUC 0.55), and human experts (AUC 0.47-0.53) for prediction of clinical responder status. It was similarly accurate in prediction of patient subjective improvement in symptoms following OBTX-A (AUC: 0.64), again outperforming other algorithms and human experts (AUC 0.41-0.62).
Conclusion:
The neural network outperformed human experts and other machine-learning approaches in prediction of objective and patient-reported OBTX-A outcomes for OAB in a challenging independent validation cohort. Clinical implementation could improve counseling and treatment selection.
