A simulation-based evaluation of machine learning models for clinical decision support: application and analysis...

Area of Science:

• Health Informatics
• Machine Learning Applications
• Clinical Workflow Optimization

Background:

• Machine learning (ML) is increasingly used in healthcare, but evaluation metrics like the c-statistic often lack clinical relevance.
• Current metrics fail to consider the integration of ML algorithms into provider workflows and their impact on patient outcomes and costs.

Purpose of the Study:

• To propose a novel model for simulating clinician use of ML algorithms within patient care pathways.
• To quantify the clinical value of ML algorithms in terms of patient outcomes and cost savings, moving beyond traditional predictive performance metrics.

Main Methods:

• Developed a simulation model to evaluate ML algorithm implementation in a clinical setting.
• Utilized data on unplanned emergency department surgical readmissions to test the model.
• Analyzed the impact of provider schedules and prediction timing on cohort size and cost-effectiveness.

Main Results:

• Provider schedules and prediction timing significantly influence the patient cohort selected by ML algorithms.
• These workflow factors critically affect the potential cost reductions achievable through readmission prevention.

Conclusions:

• Traditional ML performance metrics are insufficient for assessing clinical utility in healthcare.
• A simulation-based approach is crucial for evaluating the real-world impact of ML on patient outcomes and healthcare economics.