The heterogeneity statistic I(2) can be biased in small meta-analyses

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Meta-analysis estimates vary due to sampling error and heterogeneity.
• The I(2) statistic quantifies heterogeneity, but its accuracy is questioned in small meta-analyses.

Purpose of the Study:

• To calculate the expectation and bias of the I(2) statistic, particularly in meta-analyses with a small number of studies.

Main Methods:

• Utilized Mathematica software for precise calculations of I(2) expectation and bias.
• Focused analysis on scenarios involving a limited number of studies.

Main Results:

• I(2) exhibits significant bias in small meta-analyses.
• Bias can be positive (overestimation) with low true heterogeneity and negative (underestimation) with high true heterogeneity.
• Examples show substantial under/overestimation (e.g., 12-28 percentage points) with only 7 studies.

Conclusions:

• Interpret I(2) point estimates with caution in meta-analyses with few studies.
• Confidence intervals are recommended to supplement or replace biased I(2) estimates in small meta-analyses.