Correcting for Differences in Measurement Unreliability in Meta-Analysis of Variances

Area of Science:

• Psychometrics
• Meta-analysis
• Statistical methods

Background:

• Growing interest in meta-analytic methods for comparing variances to assess between-group variability.
• Measurement error complicates variance comparisons, as observed variance includes true score and error variance.
• Existing effect sizes like log variability ratio (lnVR) and log coefficient of variation ratio (lnCVR) may conflate true variability differences with reliability differences.

Purpose of the Study:

• To derive formulas for correcting lnVR and lnCVR and their sampling variances for differing group reliability.
• To evaluate the performance of these corrections in simulation studies.
• To provide accurate meta-analytic estimates of true-score variances when reliability varies across studies.

Main Methods:

• Derivation of formulas to adjust lnVR and lnCVR for measurement reliability.
• Calculation of corrected sampling variances for these effect sizes.
• Performance evaluation using simulation studies under conditions of varying reliability.

Main Results:

• Proposed corrections yield accurate estimates of effect sizes and sampling variances in single studies when reliability differs.
• Accurate estimation of average effect and between-study variance in random-effects meta-analysis.
• Adequate type I error rates for significance testing of the average effect.

Conclusions:

• The derived corrections effectively address between-group differences in reliability for meta-analyzing variances.
• The methods provide accurate and reliable estimates in meta-analytic research comparing group variability.
• Further research is needed to address missing or imprecise reliability data in meta-analytic datasets.