Machine Learning-Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live...

Area of Science:

• Clinical informatics
• Machine learning in healthcare
• Predictive modeling

Background:

• Machine learning models are widely used for predicting clinical risk events.
• Most models are evaluated retrospectively, with limited real-world clinical workflow and multi-hospital performance data.
• This study evaluates clinical risk prediction models in live clinical workflows across three hospitals.

Purpose of the Study:

• To evaluate clinical risk prediction models in live clinical workflows.
• To compare their performance in live settings versus retrospective data.
• To assess generalizability across three different hospitals and use cases.

Main Methods:

• Trained deep learning Transformer models for delirium, sepsis, and acute kidney injury prediction using retrospective data from three hospitals.
• Deployed models in live clinical workflows for daily practice, logging predictions against discharge diagnoses.
• Compared live performance with retrospective evaluations and conducted cross-hospital performance assessments.

Main Results:

• Model performance in live workflows closely matched retrospective data, with a slight AUROC decrease of 0.6%.
• Cross-hospital evaluations showed a significant performance drop (average AUROC decrease of 8%), emphasizing the need for local calibration.
• Performance degradation in cross-hospital tests indicates limitations of generic models.

Conclusions:

• Hospital-specific calibration of prediction models ensures good performance in live clinical settings.
• Generic machine learning models face limitations in multi-hospital deployment due to performance degradation.
• A standardized development process to create specialized models for each hospital is crucial for reliable performance across institutions.