Aspiring to clinical significance: Insights from developing and evaluating a machine learning model to predict...

Return visit admissions (RVA), which are instances where patients discharged from the emergency department (ED) rapidly return and require hospital admission, have been associated with quality issues and adverse outcomes. We developed and validated a machine learning model to predict 72-hour RVA using electronic health records (EHR) data. Study data were extracted from EHR data in 2019 from three urban EDs. The development and independent validation datasets included 62,154 patients from two EDs and 73,453 patients from one ED, respectively. Multiple machine learning algorithms were evaluated, including deep significance clustering (DICE), regularized logistic regression (LR), Gradient Boosting Decision Tree, and XGBoost. These machine learning models were also compared against an existing clinical risk score. To support clinical actionability, clinician investigators conducted manual chart reviews of the cases identified by the model. Chart reviews categorized predicted cases across index ED discharge diagnosis and RVA root cause classifications. The best-performing model achieved an AUC of 0.87 in the development site (test set) and 0.75 in the independent validation set. The model, which combined DICE and LR, boosted predictive performance while providing well-defined features. The model was relatively robust to sensitivity analyses regarding performance across age, race, and by varying predictor availability but less robust across diagnostic groups. Clinician examination demonstrated discrete model performance characteristics within clinical subtypes of RVA. This machine learning model demonstrated a strong predictive performance for 72- RVA. Despite the limited clinical actionability potentially due to model complexity, the rarity of the outcome, and variable relevance, the clinical examination offered guidance on further variable inclusion for enhanced predictive accuracy and actionability.