Related Experiment Video
Updated: Apr 26, 2026
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
From prediction to practice: early implementation of a machine learning-based hospitalization prediction tool in the
W Raven1, A de Hond2, J Vermeire1
1Department of Emergency Medicine, Leiden University Medical Centre, Albinusdreef 2, 2300 RC Leiden, the Netherlands.
Background:
Machine learning (ML) models can accurately predict hospital admissions in emergency departments (EDs), but real-world adoption remains rare. We evaluated the feasibility and early implementation of an ML-based hospitalization prediction tool in an ED of a tertiary care centre using the Medical Research Council (MRC) framework.
Methods:
A prospective mixed-methods study was conducted at the ED of the Leiden University Medical Centre (∼23,000 annual visits). A validated ML-based hospitalization prediction tool was deployed via a web-based dashboard for four months. The dashboard was designed with stakeholders' input obtained during focus groups. Staff use, perceptions, and implementation barriers were assessed through pre-/post-implementation surveys with Likert-scales and open questions, and were analyzed using Mann-Whitney U-tests. Operational outcomes (ED length of stay [LOS], hospitalization rates) were extracted from the Netherlands Emergency department Evaluation Database and compared across pre- and post-implementation periods via logistic regression adjusted for confounders.
Results:
Tool utilization was low despite an initial positive attitude towards the tool: 67% of staff reported rarely or never using it. Post-implementation surveys indicated a decline in perceived utility and reduced concern about AI replacing clinical roles. Reported barriers included lack of electronic health record integration, absence of linked actions, and misalignment with existing workflows. Hospitalization rose from 31.8% to 33.3% (p = 0.027), while ED-LOS increased during the implementation period (19.8% to 26.1%, p < 0.001); these changes could not be attributed to the tool given limited adoption.
Conclusion:
This early implementation study demonstrated low adoption of a ML prediction tool for hospitalization at the ED. Context-specific implementation barriers were identified. Guided by the MRC framework, the findings offer concrete strategies to enhance future implementation, including EHR integration, embedded action protocols, and role-specific responsibilities.
