Evaluating treatment benefit predictors using observational data: contending with identification and confounding bias

Area of Science:

• Biostatistics
• Health Informatics
• Precision Medicine

Background:

• Treatment benefit predictors (TBPs) are crucial for personalized treatment decisions in precision medicine.
• Evaluating TBP performance is challenging due to non-random treatment assignment in observational data.
• Existing evaluation methods may not adequately address confounding bias.

Purpose of the Study:

• To present a conceptual framework for evaluating pre-specified TBPs using observational data.
• To demonstrate the identification of key performance metrics (discrimination and calibration) from observable data.
• To analyze the propagation of bias in TBP evaluation under conditions of incomplete confounding control.

Main Methods:

• Review of existing TBP evaluation metrics.
• Conceptual demonstration of evaluating a TBP for binary treatment decisions at a single time point.
• Identification of population-level metrics (Concentration of Benefit Index, moderate calibration curve) using observable data distributions.
• Analysis of bias propagation in the absence of full confounding control.

Main Results:

• TBPs can be evaluated using observational data by re-expressing metrics in terms of observable data.
• Identification strategies were shown for discrimination and calibration metrics.
• Bias propagation under partial confounding is complex and often unpredictable.
• Standard intuition regarding bias direction in causal effect estimates does not apply to TBP evaluation.

Conclusions:

• Methods are presented to evaluate treatment benefit predictors using observational data, crucial for precision medicine.
• The study highlights the challenges and complexities of bias in TBP evaluation, particularly with confounding.
• The findings emphasize the need for careful consideration of confounding when assessing TBP performance in real-world data.