Using Propensity Score Analysis of Survey Data to Estimate Population Average Treatment Effects: A Case Study...

Area of Science:

• Psychological and educational research
• Survey methodology
• Statistical analysis

Background:

• Estimating population average treatment effects (PATE) from complex survey data is common in psychological and educational research.
• Propensity score methods are frequently used, with recent focus on incorporating survey weights.
• Existing literature lacks a clear summary of how survey weights impact PATE estimation.

Purpose of the Study:

• To systematically review the use of survey weights in propensity score analysis of complex survey data.
• To empirically compare PATE estimates from various analysis methods, including regression and propensity score applications.
• To assess the practical differences in PATE estimates when accounting for survey weights.

Main Methods:

• Summarized propensity score methods that incorporate survey weights.
• Applied ordinary least squares regression, weighted least squares regression, and propensity score methods to a complex survey dataset.
• Utilized the Early Childhood Longitudinal Study-Kindergarten data to estimate preschool's effect on academic achievement.

Main Results:

• The study summarized various propensity score methods for handling survey weights.
• A case study using the Early Childhood Longitudinal Study-Kindergarten data was conducted.
• Multiple analysis methods were compared for estimating PATE.

Conclusions:

• Researchers must select methods appropriate for their estimand, especially when inferring to the survey target population.
• Incorporating survey weights is crucial for accurate PATE estimation in the outcome analysis stage.
• The case study indicated limited empirical differences among the analyzed methods in a specific application.