Unsupervised Detection and Correction of Model Calibration Shift at Test-Time

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Predictive Modeling

Background:

• Clinical predictive models require generalizability and consistent performance over time.
• Model calibration shift, due to changing data distributions or prevalence, hinders model generalizability.
• Maintaining model performance across diverse healthcare systems is crucial for widespread adoption.

Purpose of the Study:

• To propose a novel method for detecting and correcting model calibration shift.
• To develop a flexible approach (CaDC) usable with any deep learning clinical model.
• To validate the method's effectiveness in improving sepsis prediction models.

Main Methods:

• Developed a model calibration detection and correction (CaDC) method.
• Utilized unlabeled data from target hospitals for calibration shift correction.
• Applied CaDC to a sepsis prediction model using three large US patient cohorts (545,089 patients).

Main Results:

• The CaDC method successfully detected and corrected calibration shift in sepsis prediction models.
• Models using CaDC achieved predefined positive predictive values (PPV) more effectively.
• For a target PPV of 20%, CaDC improved performance from 18.0% to 23.1% in external validation cohorts.

Conclusions:

• The CaDC method enhances the generalizability and reliability of clinical predictive models.
• This approach can maintain performance claims for models deployed across multiple hospital systems.
• CaDC offers a practical solution for addressing calibration drift in real-world clinical settings.