Meta-analysis of correlation coefficients: A cautionary tale on treating measurement error

Area of Science:

• Psychological Measurement
• Statistical Meta-Analysis
• Quantitative Psychology

Background:

• Measurement error is inherent in psychological scales.
• Correcting for measurement error is common practice in meta-analysis of correlations.
• Existing methods may face challenges with non-normal distributions, reliability coefficients, and ordinal data.

Purpose of the Study:

• To investigate the impact of three caveats on correcting measurement error in meta-analysis of correlations.
• To evaluate the performance of meta-analysis approaches using raw correlations and Fisher's z transformed correlations.
• To provide recommendations for improving the accuracy of meta-analytic findings.

Main Methods:

• Three simulation studies were conducted.
• Examined the effects of non-normal true score distributions.
• Assessed the influence of coefficient alpha under violated tau-equivalence assumptions.
• Investigated the impact of ordinal item scores on disattenuation formulas.

Main Results:

• Non-normal true score distributions slightly worsened estimation accuracy for mean correlations.
• Using coefficient alpha when tau-equivalence was violated led to biased mean correlation estimates and low coverage probabilities.
• Discretization of continuous items introduced bias and undercoverage, even with satisfied tau-equivalence.
• Increasing scale categories or items improved results regardless of tau-equivalence.

Conclusions:

• Researchers should be cautious when correcting for measurement error in meta-analyses.
• Violations of assumptions, particularly with coefficient alpha and ordinal data, can significantly bias results.
• Recommendations are provided to enhance the accuracy and reliability of meta-analytic correlation estimates.