Estimating Standardized Effect Sizes for Two- and Three-Level Partially Nested Data

Although previous research has discussed an effect size estimator for partially nested cluster randomized designs, the existing estimator (a) is not efficient when used with primary data, (b) can be biased when the homogeneity of variance assumption is violated, and (c) has not yet been empirically evaluated for its finite sample properties. The present paper addresses these limitations by proposing an alternative maximum likelihood estimator for obtaining standardized mean difference effect size and the corresponding sampling variance for partially nested data, as well as the variants that do not make an assumption of homogeneity of variance. The typical estimator, denoted as d (d_W with pooled SD and d_C with control arm SD), requires input of summary statistics such as observed means, variances, and the intraclass correlation, and is useful for meta-analyses and secondary data analyses; the newly proposed estimator [Formula: see text] ([Formula: see text] and [Formula: see text]) takes parameter estimates from a correctly specified multilevel model as input and is mainly of interest to researchers doing primary research. The simulation results showed that the two methods (d and [Formula: see text]) produced unbiased point and variance estimates for effect size. As expected, in general, [Formula: see text] was more efficient than d with unequal cluster sizes, especially with large average cluster size and large intraclass correlation. Furthermore, under heterogeneous variances, [Formula: see text] demonstrated a greater relative efficiency with small sample size for the unclustered control arm. Real data examples, one from a youth preventive program and one from an eating disorder intervention, were used to demonstrate the methods presented. In addition, we extend the discussion to a scenario with a three-level treatment arm and an unclustered control arm, and illustrate the procedures for effect size estimation using a hypothetical example of multiple therapy groups of clients clustered within therapists.