The sensitivity of three methods to nonnormality and unequal variances in interval estimation of effect sizes

Confidence interval (CI) estimation for an effect size (ES) provides a range of possible population ESs supported by data. In this article, we investigated the noncentral t method, Bonett's method, and the bias-corrected and accelerated (BCa) bootstrap method for constructing CIs when a standardized linear contrast of means is defined as an ES. The noncentral t method assumes normality and equal variances, Bonett's method assumes only normality, and the BCa bootstrap method makes no assumptions. We simulated data for three and four groups from a variety of populations (one normal and five nonnormals) with varied variance ratios (1, 2.25, 4, 8), population ESs (0, 0.2, 0.5, 0.8), and sample size patterns (one equal and two unequal). Results showed that the noncentral method performed the best among the three methods under the joint condition of ES = 0 and equal variances. Performance of the noncentral method was comparable to that of the other two methods under (1) equal sample size, unequal weight for each group, and the last group sampled from a leptokurtic distribution, or (2) equal sample size and equal weight for all groups, when all are sampled from a normal population, or only the last group sampled from a nonnormal distribution. In the remaining conditions, Bonett's and the BCa bootstrap methods performed better than the noncentral method. The BCa bootstrap method is the method of choice when the sample size per group is 30 or more. Findings from this study have implications for simultaneous comparisons of means and of ranked means in between- and within-subjects designs.