Related Experiment Videos
The Challenges of Predicting Rare Outcomes: A Critical Appraisal of Machine Learning Using the Pediatric
Joao Vitor Pereze de Souza1,2,3, Elizabeth M Keating3,4, William Nkenguye3,5,6
1Duke Global Health Institute, Duke University, Durham, North Carolina, USA.
Background:
Injuries are responsible for 950,000 deaths per year among children and adolescents under 18 years old. Trauma prediction scores are useful in determining severity and prognosis of injury patients. The pediatric resuscitation and trauma outcome (PRESTO) score was developed as a simple score for short-term mortality prediction in pediatric populations in low- and middle-income countries (LMICs). Using variables available at the bedside in resource-limited settings, PRESTO has been validated in South Africa, Rwanda, and Tanzania. In Tanzania, our team found these variables were present for most pediatric injury patients and that the model performed well in predicting mortality. However, we faced challenges due to small sample size and mortality being a rare outcome.
Objective:
We sought to determine whether we could improve predictability of the PRESTO model for pediatric patients by increasing sample size through inclusion of adult trauma patients from the same population at Kilimanjaro Christian Medical Center (KCMC) in Moshi, Tanzania.
Design/Methods:
Data were collected between November 2020 and February 2024 from a pediatric registry, and between April 2018 and February 2024 from an adult injury registry at KCMC. Missing data were addressed using multiple imputation. The dataset was split into training and testing sets (75/25). Ten machine learning algorithms were trained to predict in-hospital mortality using clinical and demographic variables, with performance evaluated using cross-validation, ROC-AUC, sensitivity, and specificity.
Results:
A total of 5635 injury patients (911 pediatric and 4724 adults) were included. Pediatric in-hospital mortality was 6.8%, while adult mortality was 4.4%. Pediatric patients presented with worse GCS scores compared to adults (4% severe vs. 1.4% severe). The best-performing models were Random Forest (RF), C5, and Extreme Gradient Boosting (XGB), with ROC-AUCs of 0.92, 0.90, and 0.81. In the test set, ROC-AUC for pediatric patients was lower than for adults, with differences of 0.08, 0.06, and 0.05 for XGB, RF, and C5, respectively.
Conclusions:
Adding adult data did not improve predictability of the PRESTO model, possibly due to the fact that the lower mortality and injury severity in adults compared to children in our dataset statistically diluted the model. This supports the idea that children should be considered separately, especially for vital signs and mortality prediction. These findings highlight the challenge of predicting a rare outcome, and emphasize the need to increase pediatric registry sample sizes to develop more accurate models for mortality risk stratification in LMICs.
