Validation of a prognostic model for chronic obstructive pulmonary disease (COPD) and evaluation of methods to include related data

Prognostic models estimate a patient's individual risk of future health outcomes using clinical and demographic data, supporting treatment planning. Such models are particularly useful for chronic diseases, where predicting disease progression can help guide interventions and improve patient outcomes. In chronic obstructive pulmonary disease (COPD), several models have been developed to predict long-term mortality risk. However, these models are often trained on patients diagnosed 10 to 20 years ago, and outcomes may have improved since then. As a result, such models may be miscalibrated by the time they are published, potentially limiting their clinical utility.

We aim to validate an existing COPD prognostic model for five-year mortality from the date of diagnosis to explore potential miscalibration over time and across regions. Moreover, we evaluated modelling methods to improve predictive performance in target datasets by incorporating related (ancillary) data while accounting for heterogeneity, including: (i) developing a logistic regression model in the target data, (ii) Intercept Recalibration to update the intercept using the target data, (iii) Logistic Recalibration to update both intercept and calibration slope using the target data, and (iv) Membership-based recalibration that reweights ancillary data using propensity scores. These approaches were assessed using newly collected target datasets, or by leveraging the existing development dataset only.

The existing model demonstrated similar discrimination across the external validation settings, with mean AUC values ranging from 0.79 (95% confidence interval [CI] 0.72 to 0.85) to 0.80 (95% CI 0.78 to 0.83). However, mean calibration-in-the-large (CITL) values varied from 0.001 (95% CI -0.25 to 0.34) to 0.46 (95% CI 0.29 to 0.61) in the newly available target data, likely reflecting shifts in baseline risk. Among the methods evaluated for improving prediction by incorporating ancillary data while accounting for heterogeneity, Intercept Recalibration was a simple method that generally reduced CITL to near-zero levels. Importance weighting using propensity scores also provided a potential alternative, although its performance varied across settings. These methods can be applied during development using existing data or after development with newly collected data, and may improve model calibration for the target population.