A new four-arm within-study comparison: Design, implementation, and data

Area of Science:

• Causal inference methodology
• Epidemiology
• Quantitative psychology

Background:

• Within-study comparisons (WSCs) are crucial for validating quasi-experimental designs (QEDs) by comparing their estimates to randomized controlled trials (RCTs).
• Existing WSC designs have limitations in fully assessing the internal validity of QEDs.

Purpose of the Study:

• To introduce and implement a novel WSC design to rigorously evaluate QEDs.
• To experimentally estimate the overall average treatment effect (ATE), average treatment effect on the treated (ATT), and average treatment effect on the untreated (ATU).
• To ensure sufficient statistical power for comparing QED and RCT estimates.

Main Methods:

• A new WSC design was implemented, incorporating participant preference before random assignment to enable estimation of ATE, ATT, and ATU.
• Participant recruitment and sample size (N=2200) were determined by power analyses for methodological comparisons.
• Study protocols, including eligibility criteria, recruitment, treatment allocation, and analysis, were preregistered on the Open Science Foundation, with publicly accessible data.

Main Results:

• The study successfully implemented an enhanced WSC design with a large sample size (N=2200).
• The design facilitates the estimation of multiple causal effect parameters (ATE, ATT, ATU).
• Preregistration and public data accessibility enhance transparency and reproducibility.

Conclusions:

• The developed WSC design and associated dataset provide a valuable resource for evaluating causal inference methods.
• This methodology allows researchers to test identification assumptions using real-world data.
• The study contributes to the ongoing effort to improve the validity of observational study designs.