Aligning prediction models with clinical information needs: infant sepsis case study

Sepsis recognition among infants in the Neonatal Intensive Care Unit (NICU) is challenging and delays in recognition can result in devastating consequences. Although predictive models may improve sepsis outcomes, clinical adoption has been limited. Our focus was to align model behavior with clinician information needs by developing a machine learning (ML) pipeline with two components: (1) a model to predict baseline sepsis risk and (2) a model to detect evolving (dynamic) sepsis risk due to physiologic changes. We then compared the performance of this two-component pipeline to a single model that combines all features reflecting both baseline risk and evolving risk.

We developed prediction models (two-stage pipeline and a single model) using logistic regression and XGBoost trained on electronic healthcare record data of an NICU cohort (1706 observations from 1094 patients, with a 1:1 ratio of cases to controls). We used nested 10-fold cross-validation to evaluate model performance on predictions made 1 h (T _-1) before actual clinical recognition.

The single model (XGBoost) achieved the best performance with a sensitivity of 0.77 (0.74, 0.80), specificity of 0.83 (0.80, 0.85), and positive predictive value (PPV) of 0.82 (0.79, 0.84), at 1 h prior to clinical sepsis recognition (T _-1). The pipeline model (XGBoost) achieved a sensitivity of 0.72 (0.69, 0.75), specificity of 0.84 (0.82, 0.87), and PPV of 0.82 (0.80, 0.85) at T _-1.

Our findings highlight the challenges of aligning machine learning with NICU clinical decision-making processes. The two-stage pipeline, designed to mirror clinicians' reasoning, underperformed compared to the single model. Future work should explore integrating continuous physiological data to enhance real-time risk assessment.

Although a pipeline model that separately estimates baseline and dynamic sepsis risk aligns with clinical information needs, at similar levels of specificity the observed sensitivity of the pipeline is inferior to that of a single model. Additional research is needed to better align model outputs with clinician information needs.