Related Experiment Video
Updated: Jun 1, 2026

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
Published on: September 27, 2019
The (non)comparability of the correlation effect size across different measurement procedures: a challenge to
1College of Social Work, University of Tennessee, Knoxville, USA. wnugent@utk.edu
Abstract:
Meta-analysis is becoming a principal tool for research synthesis and for the identification and justification of evidence based practices. A fundamental assumption in meta-analysis is that effect sizes based upon different measures are comparable. Recent work has challenged this assumption in the case of the standardized mean difference. In this article it is shown that population universe (true) score level correlation effect sizes, for the relationship between two constructs A and B, based upon different measures will be comparable only if construct validity invariance holds across the measures used to make inferences to A and the measures used to make inferences to B. The results of a simulation study are also reported which show that the results of a meta-analysis may be significantly and adversely affected by violations of construct validity invariance. Finally, it is concluded that the theoretical results obtained in this article, and the results of the simulation study, combine to suggest that the role of meta-analysis in the synthesis of social work research, and in the identification of evidence based practices, be de-emphasized until important questions about the sensitivity of meta-analysis to violations of construct validity invariance are answered.
Related Concept Videos
Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
Correlations
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...
One-Way ANOVA: Equal Sample Sizes
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Bias in Epidemiological Studies
Causes of Similarity-Dissimilarity Effect