The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators

Area of Science:

• Surgical Outcomes Research
• Health Informatics
• Predictive Analytics in Medicine

Background:

• Operation-specific risk calculators (OS-RCs) are often developed with the assumption of superior accuracy compared to universal risk calculators (N-RCs).
• The development and maintenance costs of OS-RCs may not be justified by marginal improvements in predictive accuracy.
• The ACS NSQIP N-RC serves as a benchmark universal risk prediction tool.

Purpose of the Study:

• To compare the predictive performance of the ACS NSQIP N-RC against operation-specific risk calculators (OS-RCs).
• To evaluate whether operation-specific predictors further enhance the accuracy of OS-RCs.
• To determine the relationship between dataset size and the relative accuracy of universal versus operation-specific models.

Main Methods:

• A large cohort of 5,020,713 NSQIP patient records were utilized, with 80% for training and 20% for validation.
• OS-RCs and OS-RCs with operation-specific predictors (OSP-RCs) were developed for six distinct operative groups and 14 outcomes.
• Predictive accuracy was assessed using Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), and Hosmer-Lemeshow P-values.

Main Results:

• The N-RC exhibited significantly greater AUROC (P=0.002) and AUPRC (P<0.001) compared to the OS-RC.
• No other statistically significant differences in accuracy were observed among the N-RC, OS-RC, and OSP-RC.
• A statistically significant inverse relationship was found between operation group sample size and the performance advantage of the N-RC over OS-RCs for both AUROC (r=-0.278, P=0.014) and AUPRC (r=-0.425, P<0.001).

Conclusions:

• The universal N-RC demonstrated superior predictive accuracy over operation-specific models, contrary to common assumptions.
• The reliance on smaller datasets for OS-RCs may limit their ability to accurately estimate predictor effects compared to the N-RC.
• While the N-RC shows advantages, the clinical significance of these differences may be negligible.