Expert-augmented machine learning for predicting extubation readiness in the pediatric intensive care unit

Determining extubation readiness in pediatric intensive care units (PICU) is challenging. We used expert-augmented machine learning (EAML), a method that combines machine learning with human expert knowledge, to predict successful extubation.

We extracted electronic health record data from patients in two PICUs. Data from patients in one unit was split into 80% training and 20% test, while patients in the other served as an external test set. EAML begins by training RuleFit, which converts gradient-boosted trees into decision rules. Then, expert clinicians were asked to assess the relative probability of successful extubation of the subgroup defined by each rule compared with the entire sample. The rules were ranked in order of increasing chance of successful extubation according to (1) the RuleFit model and (2) clinician assessment, and differences between the two rankings were calculated. The initial RuleFit model was then regularized based on these differences, producing the EAML model.

The RuleFit model selected 46 rules; we surveyed 25 clinician experts to provide feedback on them. All clinicians worked in a PICU setting and were from multiple disciplines; over half (56%) had > 5 years of PICU experience. As expected, the added regularization slightly lowered performance of EAML compared with RuleFit in the internal test set, although the difference was not statistically significant (RuleFit AUC = 0.817 vs. best-performing EAML model AUC = 0.814, difference = 0.003, 95% CI of difference = -0.009, 0.003). EAML had superior performance in the external test set (RuleFit AUC = 0.791 vs. best-performing EAML model AUC = 0.799, difference = 0.007, 95% CI of difference = 0.002, 0.013).

When creating a model to predict successful extubation in PICU patients, incorporating expert knowledge directly into the model construction process via EAML produced a model more generalizable to an external test set.