Machine learning versus traditional methods for the development of risk stratification scores: a case study using...

Artificial Intelligence and machine learning (ML) methods are promising for risk-stratification, but the added benefit over traditional statistical methods remains unclear. We compared predictive models developed using machine learning (ML) methods to the Canadian Syncope Risk Score (CSRS), a risk-tool developed with logistic regression for predicting serious adverse events (SAE) after emergency department (ED) disposition for syncope. We used the prospective multicenter cohort data collected for CSRS development at 11 Canadian EDs over an 8-year period to develop four ML models to predict 30-day SAE (death, arrhythmias, MI, structural heart disease, pulmonary embolism, hemorrhage) after ED disposition. The CSRS derivation and validation cohorts were used for training and testing, respectively, and the 43 variables used included demographics, medical history, vital signs, ECG findings, blood tests and the diagnostic impression of the emergency physician. Performance was assessed using the area under the receiver-operating-characteristics curve (AUC) and calibration curves. Of the 4030 patients in the training set and 3819 patients in the test set overall, 286 (3.6%) patients suffered 30-day SAE. The AUCs for model validation in test data were CSRS 0.902 (0.877-0.926), regularized regression 0.903 (0.877-0.928), gradient boosting 0.914 (0.894-0.934), deep neural network 0.906 (0.883-0.929), simplified gradient boosting 0.904 (0.881-0.927). The AUCs and calibration slopes for the ML models and CSRS were similar. Two ML models used fewer predictors than the CSRS but matched its performance. Overall, the ML models matched the CSRS in performance, with some models using fewer predictors.