Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding...

Area of Science:

• Health Services Research
• Biostatistics
• Public Policy

Background:

• Growing concerns exist regarding the relevance of research findings to real-world policy and practice.
• Assessing and enhancing the external validity of randomized trials is increasingly important.
• External validity determines the applicability of trial findings to specific target populations.

Purpose of the Study:

• To highlight advances in assessing and enhancing external validity of randomized trials.
• To focus on data requirements for statistical adjustments to improve applicability of findings.
• To illustrate generalization of treatment effect estimates using a case study.

Main Methods:

• A case study was used to generalize treatment effect estimates from a randomized trial sample to a target population.
• The study compared children in a Head Start supplemental program trial (Research-Based, Developmentally Informed study) to the national Head Start population (Head Start Impact Study).

Main Results:

• Generalizing findings from the trial sample to the target population was challenging.
• Limited common data elements between the trial sample and the population data were identified as a key limitation.

Conclusions:

• More publicly available data are needed to address external validity questions.
• Future studies should prioritize collecting measures comparable to other datasets.
• Enhanced measure comparability between population data and randomized trials will broaden the scope of answerable research and policy questions.