Performance of APACHE III over time in Australia and New Zealand: a retrospective cohort study

The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model has been used for benchmarking intensive care unit (ICU) outcomes in Australia and New Zealand for over a decade. This study assessed performance of the APACHE III-j model in adult patients admitted to Australasian ICUs during a ten-year period. Data were extracted from the Australian and New Zealand Intensive Care Society Adult Patient Database. Performance of APACHE III-j at different time points and within different age strata was evaluated by dividing the whole cohort into five 'two-year' groups. Calibration and discrimination were assessed by the Brier score, Hosmer-Lemeshow C and H statistics, Standardised Mortality Ratio (SMR), Cox calibration regression, calibration curves and area under the receiver operating characteristic curve (AUROC). Model performance within diagnostic categories was evaluated. Between 1 January 2000 and 31 December 2009, 558,585 ICU admissions which met inclusion criteria were included in the analysis. The mean (standard deviation) age was 60.8 (18.4) years and 58.3% were male. Overall observed mortality was 12.6%. The mean (standard deviation) APACHE III-j predicted mortality was 14.5% (21.8). Although discrimination (as measured by AUROC) was preserved over time, all other markers of model performance showed deterioration. There was a significant decrease in SMR in eight of ten most common diagnoses examined. This study demonstrates that performance of APACHE III-j model has deteriorated in Australasian hospitals and there is now a clear need for an updated modelling approach to improve mortality prediction, performance monitoring and quality of research undertaken in Australian and New Zealand ICUs.