Optimising Retraining Frequency for a Paediatric Emergency Department Admission Prediction Model: Development and...

Area of Science:

• Machine Learning in Healthcare
• Clinical Informatics
• Predictive Analytics

Background:

• Paediatric emergency departments (EDs) face challenges in predicting inpatient admissions.
• Accurate prediction is crucial for resource allocation and patient flow management.
• Temporal performance drift in predictive models necessitates regular retraining.

Purpose of the Study:

• To analyze temporal performance drift in an ensemble machine learning model for predicting paediatric ED admissions.
• To determine the optimal retraining frequency for sustained model accuracy and calibration.
• To evaluate the impact of retraining cadences on computational burden.

Main Methods:

• Utilized 409,307 ED presentations from a single tertiary paediatric hospital.
• Developed an ensemble stacking model incorporating structured triage data and BioClinicalBERT embeddings.
• Conducted a 5-year rolling-window simulation testing retraining frequencies from weekly to triennial.

Main Results:

• Weekly retraining yielded a mean AUROC of 0.843 and AMDBE of 2.57.
• Monthly retraining demonstrated non-inferior performance with significantly reduced computational cost (25% of weekly).
• Longer retraining intervals led to progressive calibration degradation and increased concept drift.

Conclusions:

• Monthly or more frequent retraining is essential for paediatric ED admission prediction models.
• Regular retraining effectively mitigates concept drift and maintains model calibration.
• This study supports the clinical implementation of regularly retrained predictive models for bed-demand forecasting.