External Validation of a Machine Learning Based Delirium Prediction Software in Clinical Routine

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• Machine learning models for clinical outcome prediction lack robust external validation and real-world performance data.
• Assessing the clinical utility of predictive software in routine practice is crucial.

Purpose of the Study:

• To deploy and prospectively evaluate a delirium prediction software in an external hospital's clinical routine.
• To compare the performance of updated and re-trained machine learning models for delirium prediction.

Main Methods:

• Machine learning models were updated and re-trained using external hospital data.
• The best-performing models were deployed for one month, with predictions compared against senior physician ratings.
• Clinician technology acceptance was assessed via a post-use questionnaire.

Main Results:

• Re-trained models exhibited high discriminative performance with an Area Under the Receiver Operating Characteristic curve (AUROC) exceeding 0.92.
• The software achieved 100.0% sensitivity and 90.6% specificity compared to clinical risk ratings.
• Users reported positive feedback on the software's usefulness, ease of use, and output quality.

Conclusions:

• Machine learning-based delirium prediction software demonstrated high discriminative performance and user acceptance in an external hospital setting.
• Retraining models with local data is effective for deploying predictive software in new clinical environments.
• The study validates the potential of AI tools for improving clinical decision-making in delirium prediction.