Does machine learning improve prediction accuracy of the Endoscopic Third Ventriculostomy Success Score? A
Armaan K Malhotra1,2, Abhaya V Kulkarni3,4, Leonard H Verhey5
1Division of Neurosurgery, Hospital for Sick Children, University of Toronto, Toronto, ON, Canada.
Purpose:
This Hydrocephalus Clinical Research Network (HCRN) study had two aims: (1) to compare the predictive performance of the original ETV Success Score (ETVSS) using logistic regression modeling with other newer machine learning models and (2) to assess whether inclusion of imaging variables improves prediction performance using machine learning models.
Methods:
We identified children undergoing first-time ETV for hydrocephalus that were enrolled prospectively at HCRN sites between 200 and 2020. The primary outcome was ETV success 6 months after index surgery. The cohort was randomly divided into training (70%) and testing (30%) datasets. The classic ETVSS variables were used for logistic regression and machine learning models. Predictive performance of each model was evaluated on the testing dataset using area under the receiver operating characteristic curve (AUROC).
Results:
There were 752 patients that underwent first time ETV, of which 185 patients (24.6%) experienced ETV failure within 6 months. For aim 1, using the classic ETVSS variables, machine learning models did not outperform logistic regression with AUROC 0.60 (95% CI: 0.52-0.69) for Naïve Bayes (highest machine learning model performance) and 0.68 (95% CI: 0.60-0.76) for logistic regression. After inclusion of imaging features (aim 2), machine learning model prediction improved but remained no better than the above logistic regression with the highest AUROC of 0.67 (95% CI: 0.59-0.75) attained using Naïve Bayes architecture compared to 0.68 (95% CI: 0.59-0.76) for logistic regression.
Conclusions:
This contemporary multicenter observational cohort study demonstrated that machine learning modeling strategies did not improve performance of the ETVSS model over logistic regression.
