External Validation Complexities: A Comparative Study of Late-Onset Sepsis Prediction Models Across Multiple Clinical...

Area of Science:

• Neonatal intensive care
• Medical informatics
• Machine learning in healthcare

Background:

• Neonatal late-onset sepsis (LOS) poses a significant threat to preterm infants in neonatal intensive care units (NICUs).
• Early detection of LOS is critical for improving infant outcomes.
• Existing data-driven prediction models for LOS face challenges with generalizability due to limited independent validation.

Purpose of the Study:

• To evaluate the performance of two distinct LOS prediction models.
• To assess the reliability of these models for clinical implementation across different settings.
• To understand the impact of external validation on model generalizability.

Main Methods:

• Two models were validated: a multi-channel feature-based extreme gradient boosting model (MC-XGB) and a deep neural network using raw RR intervals (RR-DNN).
• Validation datasets included internal (Netherlands), national external (Netherlands), and international external (U.S.) NICU data.
• Model performance was measured by the area under the receiver operating characteristic curve (AUC) across various prediction time windows.

Main Results:

• Both models achieved a peak AUC of 0.82 on the internal dataset.
• Performance declined on external datasets: RR-DNN AUCs were 0.80 (national) and 0.69 (international); MC-XGB AUCs were 0.72 (national) and 0.60 (international).
• Performance variations are likely due to differences in clinical practices, patient demographics, and monitoring technologies.

Conclusions:

• Model performance significantly decreased in external validation datasets compared to internal data.
• Implementing predictive models for LOS across diverse NICU environments presents considerable challenges.
• Standardized guidelines and enhanced data sharing are crucial for developing more robust and clinically applicable LOS prediction models.