Weighted metrics are required when evaluating the performance of prediction models in nested case-control studies

Area of Science:

• Biostatistics
• Epidemiology
• Health Informatics

Background:

• Nested case-control (NCC) designs offer efficiency for developing and validating prediction models with expensive or hard-to-obtain predictors, particularly for rare outcomes.
• Existing research primarily addresses prediction model development in NCC designs, with limited focus on robust model validation strategies.
• Accurate performance evaluation is crucial for reliable prediction models, especially in complex sampling schemes like NCC.

Purpose of the Study:

• To systematically characterize essential elements for the correct evaluation of prediction model performance within NCC data.
• To propose and illustrate methods for adjusting performance metrics to account for NCC sampling procedures.
• To compare weighted and unweighted performance metrics against full cohort data to assess bias.

Main Methods:

• Proposed adjusting standard performance metrics (C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, decision curve analysis) using sampling weights.
• Validated the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) using population-based Rotterdam study data.
• Compared performance metrics from full cohort data with those derived from NCC datasets (matched and unmatched) using both weighted and unweighted approaches.

Main Results:

• Unweighted performance metrics in NCC datasets were biased; for instance, the unweighted C-index (0.61) differed from the full cohort C-index (0.65).
• Weighted C-index (0.65) and O/E ratio (1.68) in NCC datasets closely matched full cohort values (0.65 and 1.69, respectively), demonstrating unbiased estimation.
• Weight adjustment corrected bias in threshold-based metrics and decision curve analysis, with bias increasing in matched NCC designs but still compensable by weighting.

Conclusions:

• Nested case-control studies are valuable for evaluating prediction models with complex predictors and rare outcomes.
• Performance metrics in NCC studies require adjustment using sampling weights to accurately reflect model performance.
• Weight adjustment ensures reliable and unbiased evaluation of prediction models in NCC designs, crucial for clinical utility.