Expected and empirical coverages of different methods for generating noncentral t confidence intervals for a...

Different methods have been suggested for calculating "exact" confidence intervals for a standardized mean difference using the noncentral t distributions. Two methods are provided in Hedges and Olkin (1985, "H") and Steiger and Fouladi (1997, "S"). Either method can be used with a biased estimator, d, or an unbiased estimator, g, of the population standardized mean difference (methods abbreviated Hd, Hg, Sd, and Sg). Coverages of each method were calculated from theory and estimated from simulations. Average coverages of 95% confidence intervals across a wide range of effect sizes and across sample sizes from 5 to 89 per group were always between 85 and 98% for all methods, and all were between 94 and 96% with sample sizes greater than 40 per group. The best interval estimation was the Sd method, which always produced confidence intervals close to 95% at all effect sizes and sample sizes. The next best was the Hg method, which produced consistent coverages across all effect sizes, although coverage was reduced to 93-94% at sample sizes in the range 5-15. The Hd method was worse with small sample sizes, yielding coverages as low as 86% at n = 5. The Sg method produced widely different coverages as a function of effect size when the sample size was small (93-97%). Researchers using small sample sizes are advised to use either the Steiger & Fouladi method with d or the Hedges & Olkin method with g as an interval estimation method.