Inference With Difference-in-Differences With a Small Number of Groups: A Review, Simulation Study, and Empirical...

Area of Science:

• Econometrics
• Biostatistics
• Health Services Research

Background:

• Difference-in-differences (DID) estimation is widely used for policy evaluation.
• Existing methods for handling within-group correlation in DID are not well-understood with small group numbers.
• Health research often involves longitudinal data with few groups, posing challenges for DID analysis.

Purpose of the Study:

• To review and compare statistical methods for DID estimation with small group numbers.
• To assess the performance of various corrections for within-group correlation in DID.
• To provide recommendations for robust DID analysis in health research.

Main Methods:

• Review of common DID modeling solutions: GEE, permutation tests, CSE, wild cluster bootstrapping, aggregation.
• Monte Carlo simulation study varying error correlation, group size balance, and treated group proportion.
• Empirical example using the Survey of Health, Ageing, and Retirement in Europe.

Main Results:

• Clustered standard errors (CSE) show systematic downward bias with small, unbalanced groups or few treated groups, leading to inflated Type I errors.
• Aggregation, permutation tests, bias-adjusted GEE, and wild cluster bootstrap maintain accurate coverage rates across most scenarios.
• Generalized estimating equations (GEE) may exhibit reduced statistical power.

Conclusions:

• For DID with a small number of groups, methods like aggregation, permutation tests, wild cluster bootstrap, and bias-adjusted GEE are recommended.
• These methods provide more reliable estimates compared to standard clustered standard errors.
• The findings are crucial for accurate policy impact assessment in health research with limited group data.