Causal Forests in Practice: Lessons on Detecting Heterogeneous Treatment Effects in a Randomized Controlled Trial of...

There is limited guidance on the use of causal forests in moderately sized randomized controlled trials (RCTs).

This study aimed to apply and evaluate a causal forest to estimate heterogeneous treatment effects of a healthy food subsidy program using data from a moderately sized RCT.

Using data from an RCT of the British Columbia Farmers' Market Nutrition Coupon Program (FMNCP) (n = 263), a causal forest analysis examined heterogeneous treatment effects on Healthy Eating Index-2015 scores (HEI-2015; 0-100) postintervention. Treatment effect heterogeneity was assessed in the following 3 ways: 1) using the best linear prediction test to measure covariance between predicted and true treatment effects, 2) comparisons of high versus low treatment effect groups, and 3) estimating rank-weighted average treatment effects on a targeting operator characteristics (TOC) curve and estimating the area under the TOC (AUTOC). A simulation-based power analysis examined the sample size at which the causal forest could detect a 5-point difference in HEI-2015 scores between participants with high compared with low educational attainment.

The best linear prediction test showed poor calibration and did not detect heterogeneous treatment effects (β = 0.44, P = 0.27). Heterogeneity was not detected when comparing groups with high compared with low treatment effects [-2.25, 95% confidence interval (CI): -13.53, 9.03]. The TOC curve was flat with wide CIs, and the AUTOC was not significant (-1.02; P = 0.77). Based on the simulation-based power analysis, 1050 participants were required to achieve 80% power, whereas our sample provided 40% power.

A causal forest did not detect heterogeneous treatment effects of the FMNCP on the diet quality of adults with low incomes. Simulation results indicated that the trial was underpowered, underscoring the need for larger trials. Nevertheless, with careful application and evaluation, causal forests remain a useful tool to explore heterogeneous treatment effects in moderately sized trials when the sample size is adequate. This trial was registered at clinicaltrials.gov as NCT03952338 (https://clinicaltrials.gov/ct2/home).