American College of Surgeons NSQIP Risk Calculator Accuracy Using a Machine Learning Algorithm Compared with
Yaoming Liu1, Clifford Y Ko1,2, Bruce L Hall1,3
1From the Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL (Liu, Ko, Hall, Cohen).
Journal of the American College of Surgeons
|February 2, 2023
Summary
Machine learning (ML), specifically extreme gradient boosting (XGB)-ML, offers improved surgical risk prediction accuracy compared to traditional regression models. This advancement in risk calculators (RC) enhances both discrimination and calibration for patient outcomes.
Area of Science:
- Medical Informatics
- Machine Learning in Healthcare
- Surgical Outcomes Prediction
Background:
- The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) risk calculator (RC) currently uses regression for predicting 30-day surgical outcomes.
- While regression models provide accurate risk estimates, machine learning (ML) may offer superior performance.
- This study investigates the potential of an extreme gradient boosting (XGB)-ML algorithm to enhance the accuracy of the NSQIP RC.
Purpose of the Study:
- To compare the accuracy of risk estimates generated by a regression-based approach versus an XGB-ML algorithm.
- To evaluate the performance of both methods in predicting 13 binary 30-day surgical complications and one continuous outcome (length of stay [LOS]).
Main Methods:
- A large cohort of 5,020,713 NSQIP patient records was utilized, randomly split into 80% for model development and 20% for validation.
- Risk predictions were generated using both standard regression and XGB-ML algorithms.
- Model performance was assessed using discrimination metrics (AUROC, AUPRC) and calibration statistics (Hosmer-Lemeshow, calibration curves) for binary and continuous outcomes, respectively.
Main Results:
- XGB-ML demonstrated slightly superior discrimination for binary outcomes, with higher mean AUROC (0.8299 vs 0.8251) and AUPRC (0.1558 vs 0.1476) compared to regression.
- Regression models exhibited greater miscalibration across all binary outcomes, whereas XGB-ML showed statistically significant miscalibration in only 4 out of 13 outcomes.
- For length of stay (LOS), XGB-ML achieved a lower mean squared error, indicating improved accuracy for the continuous outcome.
Conclusions:
- Extreme gradient boosting machine learning provides more accurate surgical risk predictions than traditional regression methods, particularly in terms of calibration.
- The substantial improvements in calibration suggest that transitioning the NSQIP risk calculator to an XGB-ML algorithm is warranted.
- Enhanced ML-based risk prediction can lead to more reliable assessments of patient surgical outcomes.
Related Concept Videos
01:20Actuarial Approach
108
The actuarial approach, a statistical method originally developed for life insurance risk assessment, is widely used to calculate survival rates in clinical and population studies. This method accounts for participants lost to follow-up or those who die from causes unrelated to the study, ensuring a more accurate representation of survival probabilities.
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
108
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
155
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
155
01:20Comparing the Survival Analysis of Two or More Groups
245
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...
245