External validity of four risk scores predicting 30-day mortality after surgery

Surgical risk prediction tools can facilitate shared decision-making and efficient allocation of perioperative resources. Such tools should be externally validated in target populations before implementation.

Predicted risk of 30-day mortality was retrospectively derived for surgical patients at Royal Perth Hospital from 2014 to 2021 using the Surgical Outcome Risk Tool (SORT) and the related NZRISK (n=44 031, 53 395 operations). In a sub-population (n=31 153), the Physiology and Operative Severity Score for the enumeration of Mortality (POSSUM) and the Portsmouth variant of this (P-POSSUM) were matched from the Copeland Risk Adjusted Barometer (C2-Ai, Cambridge, UK). The primary outcome was risk score discrimination of 30-day mortality as evaluated by area-under-receiver operator characteristic curve (AUROC) statistics. Calibration plots and outcomes according to risk decile and time were also explored.

All four risk scores showed high discrimination (AUROC) for 30-day mortality (SORT=0.922, NZRISK=0.909, P-POSSUM=0.893; POSSUM=0.881) but consistently over-predicted risk. SORT exhibited the best discrimination and calibration. Thresholds to denote the highest and second-highest deciles of SORT risk (>3.92% and 1.52-3.92%) captured the majority of deaths (76% and 13%, respectively) and hospital-acquired complications. Year-on-year SORT calibration performance drifted towards over-prediction, reflecting a decrease in 30-day mortality over time despite an increase in the surgical population risk.

SORT was the best performing risk score in predicting 30-day mortality after surgery. Categorising patients based on SORT into low, medium (80-90th percentile), and high risk (90-100th percentile) might guide future allocation of perioperative resources. No tools were sufficiently calibrated to support shared decision-making based on absolute predictions of risk.