A comparison of internal versus external risk-adjustment for monitoring clinical outcomes

Internal and external prognostic models can be used to calculate severity of illness adjusted mortality risks. However, it is unclear what the consequences are of using an external model instead of an internal model when monitoring an institution's clinical performance. Theoretically, using an internal prognostic model is preferred while external models are often more widely available. In this simulation study we explored the difference between the use of internal and external models on the degree and types of warning signals given by RA-EWMA control charts in the detection of increasing mortality in the ICU. Increases in mortality were correctly detected in 60% of cases (after 24 months) with the internal model, regardless of prior ICU performance. When using the external risk adjustment model, such increases were only detected for the average and poor performing ICUs. When the mortality rate was held constant, using the external model resulted in many incorrect warning signals. We conclude that the use of internal risk-adjustment models is preferable for monitoring clinical performance.