Related Experiment Video
Updated: May 7, 2026
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Optimising Retraining Frequency for a Paediatric Emergency Department Admission Prediction Model: Development and
Ethan Williams1,2, Toshi Sinha1,2, Mark Lyttle1
1Perth Children's Hospital Emergency Department, Nedlands, Australia.
Insights
Monthly retraining of machine learning models for paediatric emergency department (ED) admissions prediction is optimal. This approach mitigates concept drift and ensures accurate daily bed-demand forecasting.
Area of Science:
- Machine Learning in Healthcare
- Clinical Informatics
- Predictive Analytics
Background:
- Paediatric emergency departments (EDs) face challenges in predicting inpatient admissions.
- Accurate prediction is crucial for resource allocation and patient flow management.
- Temporal performance drift in predictive models necessitates regular retraining.
Purpose of the Study:
- To analyze temporal performance drift in an ensemble machine learning model for predicting paediatric ED admissions.
- To determine the optimal retraining frequency for sustained model accuracy and calibration.
- To evaluate the impact of retraining cadences on computational burden.
Main Methods:
- Utilized 409,307 ED presentations from a single tertiary paediatric hospital.
- Developed an ensemble stacking model incorporating structured triage data and BioClinicalBERT embeddings.
- Conducted a 5-year rolling-window simulation testing retraining frequencies from weekly to triennial.
Main Results:
- Weekly retraining yielded a mean AUROC of 0.843 and AMDBE of 2.57.
- Monthly retraining demonstrated non-inferior performance with significantly reduced computational cost (25% of weekly).
- Longer retraining intervals led to progressive calibration degradation and increased concept drift.
Conclusions:
- Monthly or more frequent retraining is essential for paediatric ED admission prediction models.
- Regular retraining effectively mitigates concept drift and maintains model calibration.
- This study supports the clinical implementation of regularly retrained predictive models for bed-demand forecasting.
Objective:
To analyse temporal performance drift and optimal retraining frequency for an ensemble machine learning model to predict inpatient admission from paediatric emergency department (ED) triage data.
Methods:
This study utilised 409,307 ED presentations from 1 July 2018 to 30 June 2024 at Perth Children's Hospital. An ensemble stacking model (XGBoost, TabNet, multi-layer perceptron and logistic regression base learners with a logistic regression meta-learner) incorporated structured triage features and tuned BioClinicalBERT-derived embeddings from free-text notes. The model ran prospectively through a 5-year rolling-window simulation, testing nine retraining cadences from weekly to triennial and a static model. Training, retraining and validation datasets were temporally separate and prior to the test set. Primary outcomes were discrimination via the area under the receiver operator characteristic (AUROC) and calibration as absolute mean daily bed error (AMDBE).
Results:
Weekly retraining achieved a mean AUROC of 0.843 (SD 0.016) and AMDBE of 2.57 (SD 1.79) over the 5-year simulation. Fortnightly and monthly cadences were non-inferior (AMDBE 2.61 and 2.73), whereas longer intervals showed progressive calibration degradation (p < 0.001) and stable AUROC. Concept drift was most pronounced in the static model, with a mean AMDBE of 10.6 in 2024 compared to 1.79 for the weekly model. Notably, monthly retraining required only 25% of the weekly computational burden with non-inferior performance.
Conclusion:
Monthly, or more frequent, model retraining sustains discrimination and calibration for paediatric ED admission prediction. This effectively mitigated concept drift and enabled accurate simulated daily bed-demand forecasting, providing evidence to support the clinical testing of such modelling.