Validating predictive models on EHRs data : lessons learnt from the NCDR-IMPACT-score applied on a French cardiac...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Prediction Models

Background:

• The IMPACT-score is a validated tool for predicting patient outcomes.
• Adapting prediction models to local patient populations is crucial for accuracy.
• Electronic Health Records (EHRs) contain valuable data for model calibration.

Purpose of the Study:

• To develop and implement a natural language processing (NLP) pipeline for extracting data from EHRs.
• To re-compute and calibrate the IMPACT-score using a pediatric hospital's specific patient cohort.
• To create a localized version of the IMPACT-score, termed IMPACT-score-Necker.

Main Methods:

• Utilized a combination of rule-based and machine learning approaches for data extraction from EHRs.
• Trained machine learning text-classifiers using an active learning annotated dataset.
• Computed the IMPACT-score using extracted data and performed logistic regression for cohort-specific calibration.

Main Results:

• Extracted clinical data from 2,980 pediatric patients.
• The localized IMPACT-score-Necker achieved an Area Under the Curve (AUC) of 0.719.
• The original IMPACT-score achieved an AUC of 0.642 in the same cohort, compared to 0.752 in its original validation cohort.

Conclusions:

• Local calibration of the IMPACT-score significantly enhanced predictive accuracy for the Necker hospital cohort.
• Model adaptation to local patient demographics and clinical variations is essential for AI tool performance.
• EHR data warehouses are vital for recalibrating and continuously monitoring AI models in clinical practice.