Analyzing and Mitigating Model Drift in Acute Kidney Injury Prediction for Hospitalized Patients.

Area of Science:

• Healthcare Informatics
• Clinical Decision Support
• Machine Learning in Medicine

Background:

• Artificial intelligence (AI) and machine learning (ML) enhance clinical risk prediction and diagnostics.
• Model performance degrades due to data drift from population changes and evolving practices.
• Predictive models for Acute Kidney Injury (AKI) are susceptible to performance decline over time.

Purpose of the Study:

• To investigate performance drift in AKI prediction models using electronic health records (EHRs).
• To evaluate the effectiveness of model updating strategies in mitigating performance drift.
• To analyze model performance across diverse patient subgroups.

Main Methods:

• Utilized EHR data from 249,749 inpatient encounters over ten years.
• Developed and applied two model updating strategies: Overall Population Update (OPU) and Specific Subgroup Update (SSU).
• Assessed model performance using the area-under-the-precision-recall-curve (AUPRC) across the overall population and nine distinct subgroups.

Main Results:

• Both OPU and SSU strategies significantly reduced performance drift.
• OPU improved average AUPRC by 0.14 overall and 0.11 across subgroups.
• SSU improved average AUPRC by 0.10 among subgroups.

Conclusions:

• Continuous model surveillance is essential in dynamic clinical settings.
• Adaptive updating strategies (OPU and SSU) are effective in maintaining reliable predictive performance.
• Regular model recalibration is necessary to counteract data drift and ensure clinical utility.