Related Experiment Videos
Development and temporal validation of a calibrated machine learning model for in-hospital blood culture positivity
Tao Huang1,2, Zhenzhen Tan2,3, Longyang Jiang1,2
1Department of Pharmacy, The Affiliated Hospital, Southwest Medical University, Luzhou, China.
Background:
Among hospitalized patients who have already undergone blood culture collection, routinely available clinical information may support early risk stratification while microbiological results are pending. We developed and temporally validated a calibrated machine learning model for in-hospital blood culture positivity.
Methods:
This single-center retrospective study included 44,389 blood culture-related hospitalization records from January 2022 to September 2024. Records from 2022 to 2023 were used for model development and records from 2024 for temporal validation. The primary outcome was at least one laboratory-recorded positive blood culture between index collection and hospital discharge. A Platt-calibrated XGBoost model using demographic characteristics, routine laboratory variables, and missingness indicators was compared with procalcitonin-only and regression-based models. Performance was assessed using discrimination, calibration, threshold-specific metrics, decision curve analysis, and SHAP.
Results:
The cohort included 3,555 outcome-positive records (8.01%). In temporal validation, the final XGBoost model achieved a ROC-AUC of 0.798 (95% CI, 0.784-0.813), a PR-AUC of 0.266, a Brier score of 0.0619, and a calibration slope of 1.061. At the 5% threshold, sensitivity was 0.877, specificity 0.570, and negative predictive value 0.983. XGBoost showed a modest improvement over logistic regression with missingness indicators, while an exploratory post hoc Age + PCT + BMI logistic comparator achieved a ROC-AUC of 0.757. Performance was stable in the new-patient validation subset but decreased within adult and pediatric subgroups (ROC-AUC 0.679 and 0.663, respectively). Age, procalcitonin, and body mass index were the dominant contributors.
Conclusion:
The calibrated model provided moderate discrimination and favorable calibration for hospitalization-level blood culture positivity risk stratification in the pooled temporal validation cohort. Performance varied across age groups, and much of the predictive information was concentrated in age, PCT, and BMI. The model does not predict the index culture result alone or diagnose clinically adjudicated bloodstream infection. Multicenter external and prospective validation using fixed prediction horizons and adjudicated outcomes is required before clinical implementation.
