From compute to care: Lessons learned from deploying an early warning system into clinical practice

Area of Science:

• Healthcare AI
• Clinical Decision Support Systems
• Machine Learning in Medicine

Background:

• Significant gap exists between developing high-performing machine learning (ML) models for healthcare and their actual clinical deployment.
• Artificial intelligence (AI) holds promise for improving healthcare, necessitating effective deployment strategies.
• CHARTwatch, an AI-based early warning system, was developed to predict patient risk of clinical deterioration.

Purpose of the Study:

• To describe the end-to-end infrastructure and process for deploying the CHARTwatch AI system.
• To identify and address challenges encountered during the deployment of a clinical AI system.
• To evaluate the success of the deployment using metrics such as model performance, workflow adherence, and infrastructure uptime.

Main Methods:

• Developed an end-to-end infrastructure for real-time data extraction and risk score communication.
• Documented technical, process-related, and pandemic-related challenges during deployment.
• Quantified deployment success through model performance, workflow adherence, and infrastructure uptime.
• Assessed adherence to Good Machine Learning Practice (GMLP) principles and identified gaps.

Main Results:

• CHARTwatch demonstrated consistent real-time performance (AUC 0.76) and strong performance on heldout test data (AUC 0.79).
• The deployment infrastructure maintained >99% uptime in the first year.
• Deployment adhered to all 10 GMLP guiding principles.
• Several crucial deployment steps, like silent testing and end-user engagement, require more detailed guidance within GMLP.

Conclusions:

• Successful deployment of the AI-based early warning system (CHARTwatch) for predicting clinical deterioration in hospitals was achieved.
• Critical success factors included meticulous data infrastructure, a silent testing phase, continuous monitoring, and strong end-user collaboration.
• Further evaluation of clinical outcomes and protocol adherence is ongoing.