Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated
Jonathan M Beus1,2, Mark Mai1,2, Nikolay P Braykov2
1Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States.
Insights
Implementing a predictive model for pediatric central line-associated bloodstream infections (CLABSI) proved challenging. Poor performance was linked to data issues and deployment hurdles, highlighting the need for better coordination.
Area of Science:
- Clinical Informatics
- Machine Learning in Healthcare
- Predictive Analytics for Infectious Diseases
Background:
- Central line-associated bloodstream infections (CLABSIs) pose significant risks to pediatric patients.
- Predictive models for CLABSI can enhance surveillance and prevention strategies.
- Previous in silico models showed promise for CLABSI prediction.
Purpose of the Study:
- To prospectively implement a pediatric CLABSI predictive model.
- To validate the model's performance for clinical practice integration.
- To identify challenges in translating predictive models to clinical settings.
Main Methods:
- Developed novel infrastructure for organizing real-time and historical patient data.
- Utilized deep learning models requiring extensive feature pre-processing.
- Compared predictive performance using two distinct CLABSI labeling methods.
Main Results:
- Model performance (AUROC) significantly decreased from 0.97 (retrospective) to <0.60 (prospective).
- Primary issues identified were train/serve skew, feature leakage, and overfitting.
- Secondary factors included complex specifications, data governance, and deployment team coordination.
Conclusions:
- Bridging the gap between model development and clinical deployment requires interdisciplinary collaboration.
- Early coordination among data governance, data science, and clinical informatics is crucial.
- Balancing predictive accuracy with implementation feasibility is key for adopting clinical decision support systems.
Abstract:
Central line-associated bloodstream infections (CLABSIs) are associated with substantial pediatric morbidity and mortality. The capacity to predict which children with central lines are at greatest risk of CLABSI could inform surveillance and prevention efforts. Our team previously published in silico predictive models for CLABSI.To prospectively implement a pediatric CLABSI predictive model and achieve adequate performance in offline validation for implementation in clinical practice.Most performant predictive models were deep learning models requiring substantial pre-processing of many features into 8-hour windows including the current day and up to 56 days prior for the current admission. To replicate this pre-processing, we created a novel infrastructure to (1) organize current-day data for all the relevant features and (2) create a staged historical data store for those same features with application programming interfaces to connect the two. We compared predictive performance of these scores for CLABSI in the next 48 hours with two labels, one based on manual review of positive blood cultures in children with central lines and another based on positive blood culture and receipt of at least 4 days of new IV antibiotics.The area under the receiver-operating characteristic (AUROC) fell from 0.97 from retrospective data to <0.60 despite multiple iterations of troubleshooting. Primary root causes included train/serve skew, feature leakage, and overfitting. Hypothesized secondary drivers were complex model specification, poor data governance, inadequate testing, challenging feature translation between real-time and historical data models, limited monitoring and logging infrastructure for troubleshooting, and suboptimal handoff between the model development and deployment teams.Bridging the gap from predictive model development to clinical deployment requires early and close coordination between data governance, data science, clinical informatics, and implementation engineers. Balancing predictive performance with implementation feasibility can accelerate the adoption of predictive clinical decision support systems.
