Concurrent external validation of bloodstream infection probability models.

Area of Science:

• Clinical Medicine
• Infectious Diseases
• Biostatistics

Background:

• Accurate prediction of bloodstream infection (BSI) is crucial for timely clinical decisions.
• Numerous multivariate models exist to estimate BSI probability.
• External validation is essential to assess model generalizability.

Purpose of the Study:

• To evaluate the real-world performance of existing BSI probability models.
• To compare the applicability, discrimination, and accuracy of these models in a single patient cohort.
• To identify common limitations affecting model utility.

Main Methods:

• Retrieved validated BSI probability models from a systematic review.
• Assessed model performance in a random sample of 4485 adult patients requiring blood cultures.
• Measured model applicability, discrimination (c-statistic), and calibration (integrated calibration index).

Main Results:

• Ten models (1991-2015) were evaluated; common issues included overfitting and variable categorization.
• Seven models had restricted applicability (<15% of patients).
• Observed discrimination was lower than reported (median c-statistic 60%), and calibration was frequently poor (median ICI 4.0%), with significant inter-model disagreement.

Conclusions:

• Published BSI probability models often exhibit limited applicability, discrimination, and calibration in external validation.
• Significant disagreement exists between models, complicating direct performance comparisons.
• Model-specific validation group dissimilarities hinder robust external validation assessments.