Interpreting treatment-effect estimates with heterogeneity and choice: simulation model results

Researchers using observational data in health-services research use various treatment-effect estimators to reduce the bias associated with unmeasured confounding variables and have focused on estimate differences to indicate the relative ability of these estimators to mitigate bias. However, available estimators may identify different treatment-effect concepts; if treatment effects are heterogeneous across patients and treatment choice reflects "sorting on the gain," then treatment-effect estimates should differ regardless of confounding. Risk-adjustment approaches yield estimates of the average treatment effect on the treated (ATT), whereas instrumental variable approaches yield estimates of a local average treatment effect (LATE).

The goal of this article was to use simulation methods to illustrate the treatment-effect concepts that are identified using observational data with various estimators.

We simulated patient treatment choices based on expected treatment valuation to observe estimates of both ATT and LATE. Different model scenarios were run to isolate the effects of both treatment-effect heterogeneity and unmeasured confounding on treatment-effect concept estimation. Models were estimated using standard linear and nonlinear estimation methods.

We show that the true values of the underlying treatment concepts differ if patients (with the help of their health care providers) make treatment choices based on expected gains, and that distinct estimators produce estimates of distinct concepts. In scenarios without unmeasured confounding, both linear and nonlinear estimation models produced estimates close to the true value of the concept identified by each estimator. However, nonlinear models suggested additional treatment-effect heterogeneity that does not exist in these scenarios.

Our results suggest that, to ensure clarity and correctness of treatment-effect estimate interpretation, it is important for researchers to state the treatment-effect concept that they are trying to identify before beginning estimation. In addition, theoretical models of treatment choice are needed to provide the foundation linking treatment-effect estimates to treatment-effect concepts and to justify instrument selection.