The Influence of Study Selection Criteria on Statistical Conclusion Validity: A Simulation Study

Randomized controlled trials (RCTs) are the gold standard for evaluating intervention effectiveness. In education research, students are often screened to identify those with reading difficulties (SWRD) who are eligible to participate. However, criteria for identifying SWRD vary widely across studies and are often shaped by practical constraints such as budget or statistical power. Common approaches include using publisher-defined risk categories, percentile-based thresholds, or selecting a fixed number of the lowest-performing students per classroom. These study-specific decisions, though necessary, may introduce unintended bias. In this simulation study, we examined how different eligibility criteria, applied to the same underlying population with a fixed "true" effect, can influence estimated intervention effects. Results showed that all tested methods produced some degree of bias in effect size estimation, but the magnitude and direction of bias were inconsistent and unpredictable. These findings suggest that variation in how researchers define SWRD can distort the field's understanding of intervention efficacy. We suggest future areas of research to guide the field toward a solution that is both statistically valid and practically feasible.