ACS NSQIP Risk Calculator Performance Across Multiple Domains of Operative Risk and Risk-associated Features

To assess the accuracy of the ACS NSQIP Risk Calculator (RC) when applied to subsets of high-risk patients.

The RC plays an especially important role in surgical decision making in situations where probabilities for adverse events are high. However, the performance of the RC in these situations has not been studied in detail. Furthermore, past criticisms of the RC have evaluated its performance when it relied on logistic regression, though it currently uses a more robust machine learning (ML) approach. This study addressed these gaps in evidence.

A sample of 1,085,707 ACS-NSQIP patients from 2021 through 2024 was partitioned into 21 subsets on the basis of various combinations of risk, age ≥ 75, ASA class 3, 4, or 5, totally dependent functional status, emergent, and laparotomy or total colectomy operative group. Calibration (absolute percentage error; APE) and discrimination (AUC) were assessed for these subsets, for mortality and morbidity outcomes, using the RC's current ML algorithm (XGB) and a potential future algorithm (CATB), which will permit risk adjustment for procedural codes beyond a single principal code as has been the traditional approach.

Across the 21 data subsets, observed mortality ranged from 0.04% to 60.01% and morbidity ranged from 3.33% to 59.17%. For both the XGB and CATB algorithms, calibration was always excellent (APE<10%) and, with few exceptions, discrimination was at least good (AUC>0.7).

Regardless of a patient population's position on the risk spectrum, the RC provides valuable information with excellent calibration and good discrimination.