Why severity models should be used with caution

There are now two validated time points for predicting hospital mortality of ICU patients--at admission and at 24 hours. The best purposes include evaluation of high clinical performance ICUs and for patients being enrolled in clinical trials. For the latter purpose, the model must be calibrated in the individual hospital to ensure that the model is applicable. This can be estimated by using goodness-of-fit testing. There are fewer uses for physiology scores and increased emphasis on converting scores to probabilities. For individual patient application, the model should be demonstrated to have high discrimination, as measured by the area under the receiver operating characteristic curve, and high calibration, as defined by goodness-of-fit testing. Although models have improved substantially and are now based on much larger databases, there is considerable uncertainty in their application for insurance purposes, triage, regulatory applications, sanctions against individual physicians, and cost containment. Current models may not adequately describe important ICU conditions such as adult respiratory distress syndrome and multi-organ dysfunction occurring after 24 hours into ICU care. For family discussions regarding prognosis of individual patients, ICU severity models must be used cautiously at admission or after 24 hours, with the understanding of the strengths and weakness of estimating probabilities of hospital mortality. The mathematical link between physiology score and estimation of hospital mortality is established only for the time point of 24 hours after ICU admission. Calibration and discrimination of the admission and 24-hour models also must be performed within each hospital in which individual probabilities are presented to families. It may be possible to customize a probability model such as MPM to achieve a high level of calibration at the individual hospital level.