Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate...

Area of Science:

• Biostatistics
• Health Services Research
• Epidemiology

Background:

• Multi-site randomized controlled trials (RCTs) are crucial for unbiased average treatment effect estimation.
• The predictive accuracy of RCTs for individual sites, particularly for informing local policy, is not well understood.

Purpose of the Study:

• To evaluate the ability of modern statistical methods to predict intervention impacts at individual sites based on multi-site RCT data.
• To assess prediction accuracy under varying levels of impact heterogeneity across sites.

Main Methods:

• Analysis of six multi-site RCTs.
• Comparison of prediction methods including lasso regression and Bayesian Additive Regression Trees (BART) against the Sample Average Treatment Effect.
• Inclusion of a wide range of moderator variables to explain impact variation.

Main Results:

• Prediction accuracy was high when impact variation across sites was minimal.
• No tested method accurately predicted impacts when substantial variation existed.
• BART generally yielded less inaccurate predictions than lasso regression or the Sample Average Treatment Effect.

Conclusions:

• Statistical modeling using typical multi-site RCT data is insufficient to explain or predict site-specific impacts when intervention effects vary considerably.
• Current prediction methods may not adequately support local policy decisions in heterogeneous settings.
• Further research is needed to improve prediction accuracy for individual sites in complex intervention studies.