Related Experiment Video
Updated: Jul 16, 2026
Adult and Pediatric Porcine Model of Acute Volume Overload
Published on: January 12, 2024
The pediatric transfer gap: How adult-ICU-trained machine-learning models degrade on pediatric intensive care
Ji-Young Yeo1, Eun Sun So2, Sungkwan Youm3
1AI Institute, Hanyang University, 222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Republic of Korea.
Insights
Clinical machine-learning models trained on adult data perform poorly in children, especially neonates. Limited pediatric data and retraining can significantly improve model accuracy and calibration for pediatric populations.
Area of Science:
- Pediatric critical care medicine
- Machine learning in healthcare
- Clinical data science
Background:
- Machine learning models trained on adult data pose safety risks when applied to pediatric populations.
- The "Pediatric Transfer Gap" quantifies performance degradation in pediatric models due to adult data training.
- This study evaluates the impact of the Pediatric Transfer Gap and the efficacy of limited target-cohort data in mitigating it.
Purpose of the Study:
- To quantify the Pediatric Transfer Gap in routinely recorded vital signs for pediatric patients.
- To assess whether limited pediatric target-cohort data can effectively close this gap.
- To evaluate the performance and calibration of machine learning models across different pediatric age bands.
Main Methods:
- Two experiments were conducted: a within-pediatric design (Experiment A) and an adult-ICU to pediatric transfer design (Experiment B).
- Experiment A used data from 10,194 pediatric intensive care patients (older children as source, younger as target).
- Experiment B utilized data from 63,203 adult ICU patients, transferring their vital signs to a pediatric cohort.
Main Results:
- Models trained on older children showed a significant drop in ROC-AUC (0.940 to 0.725) when applied to younger children, with retraining recovering performance (0.867).
- Domain shift was evident in Experiment B, primarily as severe miscalibration (ECE rising to 0.285), which was resolved by retraining.
- Model performance decreased with age, with neonates showing the lowest performance (ROC-AUC 0.587) and highest miscalibration (ECE 0.249).
Conclusions:
- Vital-sign-based mortality models trained on adult or older pediatric data lose discrimination and calibration when applied to younger children, particularly neonates.
- Target-domain retraining, even with a small amount of data (5%), significantly improves model performance and calibration.
- Pediatric-specific validation, calibration-aware reporting, and retraining are crucial before clinical implementation of these models.
Background:
Most clinical machine-learning models are trained on adult data, which raises safety concerns when they are used in children. We quantify this Pediatric Transfer Gap on routinely recorded vital signs and evaluate whether limited target-cohort data can close it.
Methods:
We performed two complementary experiments. Experiment A is a fully observed within-pediatric design in which both source and target are directly measured data from the Pediatric Intensive Care database (PIC): after item-code audit and physiological filtering, 10,194 patients yielded heart-rate, body-temperature, respiratory-rate, and oxygen-saturation summaries, with older children (≥2 years, n=3062) as source and younger children (<2 years, n=7132) as target. Experiment B is a directly observed adult-ICU design in which 63,203 adults from the MIMIC-IV ICU module supplied directly observed first-24-hour vital signs and actual in-hospital mortality, transferred to the window-matched pediatric cohort. Three classifier families (Random Forest, gradient boosting, logistic regression) were trained with stratified 70/30 splits, applied to the target without adaptation, and subjected to target-domain retraining on 50% of target data. We report five-fold cross-validation with 95% bootstrap confidence intervals for ROC-AUC and PR-AUC, Expected Calibration Error (ECE), age-band subgroup transfer, target-data learning curves, decision-curve analysis, and a suite of sensitivity analyses (age-excluded and age-normalized features; first-6/12/24-hour early windows; class-weighted training; permutation-importance explainability; and a comparison of source-only transfer, target-domain retraining, pooled training, and calibration-only updating).
Results:
The strongest discrimination evidence comes from the fully observed Experiment A: Random Forest ROC-AUC fell from 0.940 (95% CI 0.912-0.968) on older children to 0.725 (0.703-0.746) on younger children, and target-domain retraining recovered 0.867 (0.842-0.890); ECE rose from ≤0.010 to 0.142 and was restored below 0.01. The gap was essentially unchanged when age was removed as a feature (0.215 vs 0.215), excluding an age-proxy artifact, and persisted (0.152) after age-normalizing heart and respiratory rate. In the directly observed adult-ICU Experiment B the domain shift was confirmed with directly observed vitals (heart-rate Cohen's d=+2.75, respiratory-rate d=+3.06) and manifested mainly as severe miscalibration (ECE rising to 0.285) with a smaller, model-dependent discrimination change, again removed by target-domain retraining. Performance fell monotonically with age, from 0.992 in preschool children to 0.587 in neonates (ECE 0.249). Retraining on only 5% of the available target pool (≈180 patients) recovered most of the gap; calibration-only updating corrected ECE but could not restore discrimination.
Conclusions:
Vital-sign-based mortality models trained on older children or directly observed adult-ICU data lose discrimination and become severely miscalibrated when applied to younger children, with neonates worst affected. The fully observed within-pediatric experiment provides the primary physiological evidence; the directly observed adult-ICU experiment is supportive and shows that the shift is dominated by miscalibration. Before clinical implementation, such models require pediatric- and age-band-specific validation, calibration-aware reporting, and a modest amount of target-domain retraining.