Predicting disease progression under a "treatment-naive" counterfactual scenario in a competing risks framework

Prognostic models aim to estimate the individual risk of developing a specific health outcome based on patient characteristics and prognostic factors. A key factor affecting performance over time and across different populations is evolving treatment strategies. These shifts can alter the relationship between predictors and outcome, as well as occurrence of competing events. Counterfactual risk models may yield more reliable estimates across different time periods and treatment regimens than traditional approaches like ignoring treatment, restricting analysis to untreated patients, stratifying, or adjusting for treatment.

In the Turin Prostate Cancer Prognostication cohort study, which includes patients diagnosed with non-metastatic prostate cancer and followed longitudinally, the performance of Memorial Sloan Kettering Cancer Center prediction model was evaluated in predicting metastatic progression up to 5 years after diagnosis in the presence of competing mortality risk. Using the g-formula, time to the two competing events was generated under a counterfactual scenario in which no patients received prostatectomy. The performance of the MSKCC model developed in the TPCP cohort and applied to the counterfactual cohort (applied model) and the model updated under this counterfactual scenario (updated model) were compared.

At 5 years post-diagnosis, the updated model had an Area Under the Curve (AUC) of 0.84 (95% CI: 0.82; 0.85) and the applied model had an AUC of 0.83 (95% CI: 0.81, 0.85). The calibration slope for the updated model was 1.02 (95% CI: 0.99, 1.12) compared to 1.07 (95% CI: 0.97, 1.17) for the applied model. Calibration intercepts were 0.00 (95% CI: -0.10, 0.10) and 0.05 (95% CI: -0.04, 0.15) respectively. The predicted probabilities of the outcome of interest under counterfactual treatment strategy varied not only because of the treatment itself, but also due to its impact on competing risks.

When a prediction model cannot be validated in a new population due to treatment shifts, counterfactual risks under hypothetical regimens can still be cautiously estimated. In particular, counterfactual modeling should be considered as a useful complementary assessment of model transportability and calibration under clinically relevant treatment scenarios.