External validation of prognostic models for critically ill patients required substantial sample sizes

Area of Science:

• Critical Care Medicine
• Health Services Research
• Biostatistics

Background:

• External validation of prognostic models is crucial for assessing their real-world performance.
• Commonly used performance measures may behave unpredictably when applied to smaller datasets during validation.

Purpose of the Study:

• To evaluate the behavior of predictive performance measures used in external validation of intensive care unit (ICU) prognostic models.
• To determine the impact of sample size on the reliability of these performance measures.

Main Methods:

• Four established prognostic models were assessed using the Dutch National Intensive Care Evaluation registry (n=41,239).
• Model discrimination (AUC) and accuracy (Brier score), alongside calibration measures, were evaluated on the full dataset and random subsamples.
• Performance metrics from subsamples were compared to those from the complete dataset.

Main Results:

• Model performance differences were minimal across the evaluated prognostic models.
• Area Under the Curve (AUC) and Brier scores exhibited significant variability with smaller sample sizes.
• Calibration measures were highly sensitive to sample size, and statistical power to detect performance differences was low.
• Direct performance comparisons without statistical analysis proved unreliable.

Conclusions:

• Substantial sample sizes are essential for robust external validation and reliable model comparison.
• Calibration statistics and significance testing are not recommended for performance assessment with limited data.
• A simple customization approach is advised to address model lack-of-fit issues in external validation.