Correcting for case-mix shift when developing clinical prediction models

Area of Science:

• Clinical Epidemiology
• Health Informatics
• Biostatistics

Background:

• Clinical prediction models (CPMs) can be affected by case-mix shift, where predictor distributions change in development datasets.
• This shift can impact model performance during deployment.
• This study leverages observed case-mix shifts within development data to address deployment-phase shifts.

Purpose of the Study:

• To introduce and evaluate a novel Membership-based method for correcting case-mix shift during CPM development.
• To assess the impact of this method on CPM predictive performance under various shift and sample size scenarios.

Main Methods:

• A Membership-based method using a probabilistic similarity metric to re-weight source data samples.
• Application to a real-world dataset of myocardial infarction patients with out-of-hospital cardiac arrest.
• Evaluation across nine scenarios, comparing the proposed method against models ignoring shift or using only recent data.

Main Results:

• The Membership-based method shows promise, particularly with insufficient target set sample sizes, achieving an optimism-adjusted calibration slope (c-slope) of 0.98 in partial shift scenarios.
• When target sample size was sufficient, the Unweighted model on target data only performed better (c-slope 0.95) than the Membership-based model (c-slope 0.92).
• In complete case-mix shift scenarios, both Membership-based and Unweighted models performed similarly, achieving c-slopes of 0.77 (insufficient target data) and 0.94 (sufficient target data).

Conclusions:

• The Membership-based method is a promising approach for addressing case-mix shift in CPM development, especially when target data is limited.
• Further research is needed to validate the method and explore its effectiveness with other types of data distribution shifts.
• Optimizing CPMs for evolving data distributions is crucial for maintaining predictive accuracy.