Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data.

Area of Science:

• Biostatistics
• Epidemiology
• Health Informatics

Background:

• Evaluating predictive models for medical decision-making is crucial.
• Standard performance evaluation methods are inadequate for observational data under interventions.
• Predictions under interventions estimate outcomes if a specific treatment strategy were followed.

Purpose of the Study:

• To develop and evaluate methods for assessing the counterfactual performance of predictions under interventions for time-to-event outcomes.
• To adapt standard performance measures for counterfactual evaluation using longitudinal observational data.
• To support informed medical decision-making by quantifying prediction performance under hypothetical treatment strategies.

Main Methods:

• Utilized artificial censoring and inverse probability weighting to create a validation dataset mimicking the intervention strategy.
• Extended established performance metrics including calibration, discrimination (c-index, cumulative/dynamic AUCt), and Brier score.
• Employed a simulation study to validate the proposed methods and assess their ability to detect poor performance.

Main Results:

• The proposed methods successfully evaluate counterfactual performance for time-to-event predictions under interventions.
• Extended metrics demonstrated effectiveness in assessing calibration, discrimination, and overall prediction error.
• The approach was successfully applied to liver transplantation data, quantifying prediction performance for organ allocation decisions.

Conclusions:

• This work provides a robust framework for evaluating counterfactual prediction performance using observational data.
• The developed methods enhance the reliability of predictions used in medical decision-making, particularly for time-to-event outcomes.
• Quantifying prediction performance under interventions is essential for critical applications like organ allocation.