Evaluation of Sequential Organ Failure Assessment-2 Performance Across Demographic Subgroups: An External Validation Using MIMIC-IV
Related Experiment Videos
Jacob Ellen1,2, Sicheng Hao2,3, Catherine A Gao4
1Harvard Medical School, Boston, MA.
Objectives:
To evaluate the discrimination and calibration of the Sequential Organ Failure Assessment (SOFA)-2 score for ICU mortality across demographic subgroups defined by age, sex, race and ethnicity, primary language, and insurance status.
Design:
Retrospective cohort study.
Setting:
Single academic medical center in the northeastern United States (Beth Israel Deaconess Medical Center, Boston, MA), using the Medical Information Mart for Intensive Care-IV database (version 3.1), 2008-2022, a dataset external to the SOFA-2 development cohort.
Patients:
Adult patients (≥ 18 yr) with at least one ICU admission. The first ICU admission of the first qualifying hospitalization per patient was analyzed.
Interventions:
None.
Measurements And Main Results:
First-day SOFA-2 scores were calculated from the worst values recorded during the first 24 hours of ICU admission across six organ systems, with ICU mortality as the primary outcome. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC) and calibration using intercept and slope, with subgroup differences in AUROC estimated by bootstrap resampling. Among 64,015 ICU patients (mean [sd] age, 64.6 [17.1] yr; 56.1% male; 66.1% White), overall AUROC was 0.77 (95% CI, 0.76-0.77). Discrimination declined significantly with age: AUROC, 0.85 (0.83-0.87) for ages 18-44 years vs. 0.72 (0.70-0.73) for ages 75 or older (ΔAUROC, -0.14 [95% CI, -0.16 to -0.11]), with systematic underprediction in older patients. Discrimination was significantly decreased among non-English speakers (ΔAUROC, -0.04) but did not differ across documented racial/ethnic groups. Patients with missing demographic information had markedly higher mortality than the overall cohort rate of 7.2% (14.1% for missing race/ethnicity; 23.1% for missing language).
Conclusions:
SOFA-2 demonstrated good overall performance but with clinically meaningful variation across subgroups, particularly with age and language. Patients with missing demographic data experienced markedly higher mortality. These findings argue for routine equity evaluation as a complement to traditional validation of clinical prediction tools.