Simpson's paradox in the integrated discrimination improvement

Area of Science:

• Biostatistics
• Medical Informatics
• Epidemiology

Background:

• The integrated discrimination improvement (IDI) is a key metric for comparing risk prediction models.
• The IDI quantifies how a new model improves risk prediction by increasing risk for events and decreasing it for non-events.
• However, the IDI is susceptible to Simpson's paradox, potentially leading to misleading conclusions.

Purpose of the Study:

• To investigate the impact of imbalanced covariates on the IDI and other performance metrics.
• To provide an analytic explanation for the Simpson's paradox observed in the IDI.
• To introduce and evaluate a Weighted IDI as a more robust metric.

Main Methods:

• Extensive simulations were conducted using various performance metrics, including IDI, AUC, Brier score, and R².
• An analytic framework was developed to explain the paradox in IDI.
• The Cancer Genomics Network data was used to illustrate the paradox with a breast and ovarian cancer risk prediction model (BRCAPRO).

Main Results:

• Simpson's paradox in IDI consistently occurred under stratum-specific mis-calibration.
• Overall model calibration did not ensure stratum-level calibration in simulations or real-world data.
• The Weighted IDI provided better insights into the paradox compared to the standard IDI.

Conclusions:

• The IDI should be used cautiously, particularly with imbalanced covariates, and only when stratum-level calibration is met.
• Overall model calibration is insufficient to guarantee reliable performance across all strata.
• Additional metrics are recommended to validate IDI findings and ensure accurate risk prediction model assessment.