Development and temporal validation of machine learning models for predicting clinically relevant medication...

Medication discrepancies at hospital admission are common and can cause preventable patient harm. Predictive models can help prioritize medication reconciliation for high-risk patients. This study aimed to develop and validate machine learning (ML) models for predicting clinically relevant medication reconciliation discrepancies in emergency department (ED) patients, and to compare their performance with logistic regression.

We conducted a single-center, retrospective study at UZ Leuven. The dataset included patients admitted to the ED between 2017 and 2019 (development set) and 2021-2022 (temporal validation set). The outcome variable was the presence of at least one clinically relevant medication discrepancy, defined by expert panel adjudication. Variables were extracted from the electronic health record, with care to avoid data leakage. Three models - logistic regression, random forest, and eXtreme Gradient Boosting - were developed using tailored variable selection strategies, and validated temporally. Model performance was assessed via discrimination, calibration, and classification metrics. Clinical utility was assessed using decision curve analysis.

The development and validation cohorts included 817 and 349 patients, respectively. LR and RF models demonstrated moderate discrimination on temporal validation (AUROC 0.67-0.68). The XGBoost model showed lower discrimination (AUROC 0.63). Calibration was comparable across models. Decision curve analysis showed only small differences in net benefit between models across clinically relevant threshold probabilities.

ML models provided no clear improvement over logistic regression, which achieved similar predictive performance and greater interpretability. These findings highlight both the potential and the limitations of ML for supporting targeted medication reconciliation in ED workflows. Future research should explore the added value of richer data sources, such as unstructured clinical narratives.