Examining the normality assumption of a design-comparable effect size in single-case designs

Area of Science:

• Single-case experimental design (SCED) and meta-analysis.
• Statistical methodology for intervention effect size estimation.

Background:

• The design-comparable effect size (gAB) is recommended by the What Works Clearinghouse (WWC) for SCED studies and meta-analysis.
• Previous research has not evaluated gAB's performance with non-normal data distributions.

Approach:

• Expanded upon Pustejovsky et al. (2014) to investigate gAB performance in multiple-baseline (MB) designs.
• Examined the influence of data distributions, number of cases (m), measurements (N), intra-class correlation (ρ), variance ratio (λ), and autocorrelation (ϕ).
• Assessed performance using relative bias (RB), relative bias of variance (RBV), mean squared error (MSE), and confidence interval coverage rate (CR).

Key Points:

• gAB remains unbiased even with non-normal data distributions.
• Variance of gAB is generally overestimated, leading to wider confidence intervals, particularly with normal distributions and small m and N.
• Data distributions significantly impact RB (49% variance), while m and ρ substantially influence MSE (34% variance each).

Conclusions:

• Recommend gAB for MB studies and meta-analysis with N ≥ 16, under specific conditions of data distribution normality, m, and ρ.
• Suggest guidelines for optimal gAB application based on data characteristics and sample size parameters.
• Emphasize the importance of understanding gAB's applicability, design-comparability, and transparent reporting practices for effect size indices.