Detecting changes in the performance of a clinical machine learning tool over time.

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Diagnostic Stewardship

Background:

• Overutilization of blood cultures (BCs) in Emergency Departments (EDs) leads to low diagnostic yield and high contamination rates.
• This contributes to increased antibiotic use and unnecessary diagnostic procedures.
• A previously developed machine learning (ML) model aimed to predict BC outcomes and improve diagnostic stewardship.

Purpose of the Study:

• To conduct a real-time evaluation of the ML model's performance in predicting blood culture outcomes.
• To assess the model's stability and identify potential performance drift over time.
• To determine the need for recalibration or correction of the BC stewardship tool.

Main Methods:

• The ML model was integrated into the Electronic Health Record system for real-time prediction of BC outcomes in adult ED patients.
• Model performance was monitored monthly using Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores.
• Statistical Process Control (SPC) charts were employed to track performance variations and detect drift.

Main Results:

• The model achieved an average AUC of 0.78, AUPRC of 0.41, and Brier score of 0.10 across 3,035 patient visits.
• Despite changes in patient demographics and clinical practices, SPC charts indicated stable model performance with no points outside the control range.
• The average blood culture positivity rate was 13.4% during the evaluation period.

Conclusions:

• The BC stewardship tool demonstrated robust and stable performance, indicating resilience to evolving clinical environments.
• SPC charts provide an effective method for monitoring model performance and detecting drift.
• No recalibration or correction of the BC stewardship tool was necessary during the study period.