Developing well-calibrated illness severity scores for decision support in the critically ill

Area of Science:

• Critical Care Medicine
• Health Informatics
• Biostatistics

Background:

• Illness severity scores are crucial for ICU quality improvement and benchmarking.
• Current models exhibit poor generalization, especially in probability calibration for high-risk patients, limiting decision support utility.
• Existing models underperform for patients with high mortality risk.

Purpose of the Study:

• To evaluate a sequential modeling approach for improving illness severity score calibration in ICUs.
• To compare the sequential approach against logistic regression and gradient boosting machine models.
• To enhance the reliability of risk prediction across the full spectrum of patient risk.

Main Methods:

• A sequential modeling strategy was developed: an initial regression model for general risk assignment, followed by a high-risk-specific model.
• This approach was compared with standard logistic regression and a gradient boosting machine (GBM).
• Model performance was assessed using receiver operating characteristic (ROC) curves, precision-recall curves, and reliability curves.

Main Results:

• The sequential modeling approach demonstrated improved reliability curves compared to baseline models.
• No significant impact on receiver operating characteristic (ROC) or precision-recall curves was observed with the sequential method.
• The gradient boosting machine (GBM) showed a marginal improvement in discrimination but similar calibration to sequential models.

Conclusions:

• The sequential modeling approach enhances the reliability of illness severity scores, particularly for high-risk ICU patients.
• This improved calibration offers better potential for clinical decision support in intensive care settings.
• Gradient boosting machines offer slight discrimination benefits, with comparable calibration to the proposed sequential method.