External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients

Numerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data.

Patient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis.

After assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC<0.6), 39 (45%) moderate (AUC:0.6-0.7), 38 (46%) good (AUC:0.7-0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models.

Comprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.