Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials

Area of Science:

• Econometrics
• Causal Inference
• Program Evaluation

Background:

• The 2003 Bloom, Hill, and Riccio (BHR) paper provided foundational methods for analyzing site-level mediators in multi-site randomized trials.
• Existing methods often rely on aggregate site-level data, potentially limiting precision and introducing bias.

Purpose of the Study:

• To enhance methods for analyzing multi-site randomized trials by incorporating student-level data for mediator and confounder measurement.
• To improve the accuracy and robustness of causal inference in socio-economic intervention evaluations.

Main Methods:

• Developed new statistical methods utilizing student-level data to estimate site-level mediators and confounders.
• Employed asymptotic behavior analysis, simulations, and an empirical application to assess method performance.
• Examined bias, mean square error, and confidence interval coverage for mediation coefficient estimates.

Main Results:

• Simulations indicate the proposed methods generally improve inference quality, even without confounding.
• Empirical analysis of the Health Professions Opportunity Grants (HPOG) Program revealed significant mediation effects.
• Program-average full-time equivalent (FTE) months of study by month six mediated career progress and degree receipt.

Conclusions:

• The proposed student-level data methods offer a robust approach for BHR-style analyses in program evaluations.
• These methods enhance the precision and reliability of estimating mediation effects in multi-site trials.
• Findings from the HPOG program demonstrate the practical utility of these enhanced analytical techniques.