A simulation study comparing properties of heterogeneity measures in meta-analyses

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Assessing heterogeneity (between-study variance) is vital in meta-analysis, influencing statistical methods and result interpretation.
• Traditional heterogeneity tests can be misleading due to low power or detecting irrelevant heterogeneity, potentially leading to inappropriate models or complex analyses.

Purpose of the Study:

• To review and compare two heterogeneity measures with different scaling against the standard heterogeneity test.
• To introduce a new total information measure for heterogeneity assessment.
• To discuss estimates of within-study variance and their impact on meta-analysis.

Main Methods:

• A simulation study was conducted to assess various properties of heterogeneity measures and tests.
• Comparison of two distinct heterogeneity measures and the standard heterogeneity test.
• Evaluation of within-study variance estimation methods, including basing estimates on squared weights versus the sum of weights.

Main Results:

• Heterogeneity tests and measures quantify the relative increase in variance due to heterogeneity, not the absolute amount.
• Estimating within-study variance using squared weights of individual studies is more favorable than using the sum of weights.
• A heterogeneity measure scaled to a fixed interval requires reference values for interpretation, while a measure based on the between- to within-study variance ratio offers a more natural interpretation.

Conclusions:

• Heterogeneity measures are more suitable than tests for determining appropriate meta-analysis strategies.
• Both reviewed measures quantify the impact of heterogeneity on meta-analysis outcomes, influenced by individual study effect variances and sample sizes.
• The choice of within-study variance estimation and the interpretation of heterogeneity measures are critical for robust meta-analysis.