Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated...

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Predictive Analytics for Infectious Diseases

Background:

• Central line-associated bloodstream infections (CLABSIs) pose significant risks to pediatric patients.
• Predictive models for CLABSI can enhance surveillance and prevention strategies.
• Previous in silico models showed promise for CLABSI prediction.

Purpose of the Study:

• To prospectively implement a pediatric CLABSI predictive model.
• To validate the model's performance for clinical practice integration.
• To identify challenges in translating predictive models to clinical settings.

Main Methods:

• Developed novel infrastructure for organizing real-time and historical patient data.
• Utilized deep learning models requiring extensive feature pre-processing.
• Compared predictive performance using two distinct CLABSI labeling methods.

Main Results:

• Model performance (AUROC) significantly decreased from 0.97 (retrospective) to <0.60 (prospective).
• Primary issues identified were train/serve skew, feature leakage, and overfitting.
• Secondary factors included complex specifications, data governance, and deployment team coordination.

Conclusions:

• Bridging the gap between model development and clinical deployment requires interdisciplinary collaboration.
• Early coordination among data governance, data science, and clinical informatics is crucial.
• Balancing predictive accuracy with implementation feasibility is key for adopting clinical decision support systems.