Prospective validation and real-time implementation of an automated machine learning postoperative mortality...

Area of Science:

• Medical Informatics
• Machine Learning in Healthcare
• Surgical Outcomes Research

Background:

• Prospective validation is crucial for machine learning (ML) prediction models to ensure implementation fidelity and feasibility.
• A previously reported ML model for predicting postoperative mortality in surgical inpatients required validation.
• Evaluating the feasibility of a pilot clinical decision support tool was a secondary objective.

Purpose of the Study:

• To prospectively validate a machine learning model for predicting in-hospital mortality in surgical patients.
• To assess the feasibility of integrating this ML model into clinical practice via a pilot clinical decision support tool.

Main Methods:

• A random forest ML model for in-hospital mortality prediction was prospectively validated and implemented.
• The model was integrated into the electronic health record (EHR) using a real-time data mart.
• Model performance was assessed using area under the receiver operating characteristic curve (AUROC) and area under the curve precision-recall (AUCPR).
• Feasibility was evaluated through workflow metrics and a survey of anesthesiologists.

Main Results:

• The prospectively implemented model achieved an AUROC of 0.874 (95% CI 0.860-0.887) and an AUCPR of 0.111.
• The original 58-feature model had an AUROC of 0.925, and ASA physical status had an AUROC of 0.814.
• Implementation feasibility was confirmed by real-time data updates, automated EHR output transfer, and positive provider feedback.

Conclusions:

• Prospective validation and EHR implementation of the ML mortality prediction model showed acceptable real-world performance.
• The study demonstrated the feasibility of integrating such ML-based decision support systems into routine clinical practice.
• This approach holds promise for improving the prediction and management of postoperative mortality.