Evaluating Discrimination of ACS-NSQIP Surgical Risk Calculator in Thyroidectomy Patients

The ACS-NSQIP surgical risk calculator (SRC) often guides preoperative counseling, but the rarity of complications in certain populations causes class imbalance, complicating risk prediction. We aimed to compare the performance of the ACS-NSQIP SRC to other classical machine learning algorithms trained on NSQIP data, and to demonstrate challenges and strategies in predicting such rare events.

Data from the NSQIP thyroidectomy module ys 2016 - 2018 were used to train logistic regression, Ridge regression and Random Forest classifiers for predicting 2 different composite outcomes of surgical risk (systemic and thyroidectomy-specific). We implemented techniques to address imbalanced class sizes and reported the area under the receiver operating characteristic (AUC) for each classifier including the ACS-NSQIP SRC, along with sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) at a 5% - 15% predicted risk threshold.

Of 18,078 included patients, 405 (2.24%) patients suffered systemic complications and 1670 (9.24%) thyroidectomy-specific complications. Logistic regression performed best for predicting systemic complication risk (AUC 0.723 [0.658 - 0.778]); Random Forest with RUSBoost performed best for predicting thyroidectomy-specific complication risk (0.702; 0.674 - 0.726). The addition of optimizations for class imbalance improved performance for all classifiers.

Complications are rare after thyroidectomy even when considered as composite outcomes, and class imbalance poses a challenge in surgical risk prediction. Using the SRC as a classifier where intervention occurs above a certain validated threshold, rather than citing the numeric estimates of complication risk, should be considered in low-risk patients.