Do population-level risk prediction models that use routinely collected health data reliably predict individual...

Area of Science:

• Health Informatics
• Epidemiology
• Biostatistics

Background:

• Routinely collected health data (RCD) are increasingly used for clinical research and risk prediction.
• Heterogeneity in data recording and patient populations across clinical sites poses challenges for reliable risk prediction.
• Existing risk prediction models may not fully account for site-specific variations.

Purpose of the Study:

• To assess the reliability of individual risk predictions derived from RCD, considering inter-site heterogeneity.
• To evaluate the performance of a random effects model compared to standard QRISK3 predictions for cardiovascular disease (CVD).

Main Methods:

• Utilized a large dataset of 3.6 million patients from 392 sites in the Clinical Practice Research Datalink.
• Employed Cox models incorporating QRISK3 predictors and a frailty (random effect) term for each site to address unmeasured site variability.
• Analyzed variations in data recording (e.g., BMI missingness) and CVD incidence rates across general practices.

Main Results:

• Significant variation observed in data recording completeness (BMI missingness: 18.7%-60.1%) and CVD incidence rates (0.4-1.3 per 100 patient-years) between practices.
• Individual CVD risk predictions from the random effects model showed inconsistency with QRISK3 predictions (e.g., 10% QRISK3 risk had a 95% range of 7.2%-13.7% with random effects).
• Random variability accounted for only a small portion of the observed prediction inconsistency; however, the random effects model demonstrated equivalent discrimination and calibration to QRISK3.

Conclusions:

• Risk prediction models using RCD perform adequately at a population level but exhibit considerable uncertainty for individual predictions.
• The heterogeneity between clinical sites in data and populations significantly affects the reliability of individual risk estimates.
• Clinicians and patients require clear communication regarding the inherent uncertainty in individual risk predictions derived from routinely collected data.