Distributional Analysis in Educational Evaluation: A Case Study from the New York City Voucher Program

Area of Science:

• Education Policy
• Econometrics
• Educational Psychology

Background:

• School choice programs are a key policy lever aimed at improving student outcomes.
• Previous research often focuses on average treatment effects, potentially masking varied impacts across student subgroups.
• Understanding program effects across the entire achievement distribution is crucial for equitable educational policy.

Purpose of the Study:

• To estimate the causal effects of the New York City School Choice Scholarship Program (NYCSCSP) on student achievement using quantile treatment effects.
• To demonstrate the utility of distributional effects estimation in educational research beyond mean-effect analyses.
• To explore the heterogeneity of educational treatment effects and their implications for intervention justification and external validity.

Main Methods:

• Utilized quantile treatment effects (QTE) estimation.
• Analyzed data from a random-assignment study of the NYCSCSP.
• Examined effects across the full distribution of student achievement, not just the mean.

Main Results:

• The NYCSCSP demonstrated negligible and statistically insignificant effects on student achievement across all quantiles.
• No significant positive or negative impacts were detected at any point in the achievement distribution.
• The findings suggest the program did not substantially alter students' relative standing in academic performance.

Conclusions:

• Distributional effects estimation provides a richer understanding of educational interventions than traditional mean-effect analyses.
• This methodology can reveal nuanced impacts and inform hypotheses about treatment effect heterogeneity.
• QTE analysis is valuable for assessing the external validity of interventions by situating effects within national achievement distributions.