American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator Accuracy When...

To determine whether American College of Surgeons National Surgical Quality Improvement Program risk calculator (RC) accuracy can be improved by incorporating Current Procedural Terminology (CPT) codes beyond the principal code.

Because of technical limitations, past and current RC algorithms have relied only on the principal CPT code, represented as a logit score, to adjust for procedure-related risk. RC performance was evaluated when using a new machine learning (ML) algorithm capable of incorporating an indeterminate number of high cardinality categorical variables (in this case, multiple CPT codes).

American College of Surgeons National Surgical Quality Improvement Program data from 5,020,713 patients from 2016 to 2020 were used. Predictive accuracy, for 13 outcomes, was assessed when the RC relied on, in addition to standard predictors, a logit score associated with the principal CPT code [extreme gradient boosting (XGB) ML], or up to 21 codes in native categorical form [CatBoost (CATB) ML]. Of the cases, 80% were used for training and 20% for validation. Discrimination (area under the receiver operator characteristic curve and area under the precision recall curve) and calibration (Hosmer-Lemeshow statistics) were assessed on the entire validation data set and on a subset of that data that included only patients who had at least 1 CPT code recorded beyond the principal code.

There was no consistent accuracy advantage of CATB over XGB with respect to discrimination. XGB tended to have slightly better calibration than CATB when evaluated on the complete validation data set but tended to have slightly worse calibration compared with CATB when the validation data set was limited to the subset of 34.8% of cases where there was at least one code in addition to the principal CPT code. However, there was a subset of patients with 4 or more CPTs (about 8% of all patients) where CATB provided meaningfully more accurate estimates than XGB.

While the current RC, relying on XGB and the principal CPT code, remains a viable approach to routine surgical risk assessment, an advanced version of the RC, based on the CATB algorithm and accommodating multiple CPT codes, may provide more accurate estimates.