Small studies are more heterogeneous than large ones: a meta-meta-analysis

Area of Science:

• Biostatistics
• Clinical Epidemiology
• Medical Research Methodology

Background:

• Between-study heterogeneity is a key component of random-effects models in meta-analysis.
• Small clinical trials often exhibit larger effect sizes, prompting investigation into their relationship with heterogeneity.

Purpose of the Study:

• To empirically evaluate the relationship between trial size and heterogeneity (τ) in meta-analyses.
• To determine if smaller studies contribute disproportionately to heterogeneity within meta-analyses.

Main Methods:

• Analysis of 2,009 dichotomous and 1,254 continuous outcome meta-analyses from the Cochrane Database of Systematic Reviews (2009-2013).
• Evaluation of the association between estimated τ and trial size using regression across meta-analyses.
• Within-meta-analysis evaluation using a Bayesian approach, defining small trials as those with standard errors (SEs) over 0.2 standardized effects.

Main Results:

• Small studies demonstrated larger heterogeneity (τS(2)) compared to larger studies within the same meta-analysis (average ratio 2.11 for dichotomous, 3.11 for continuous outcomes).
• The imprecision in estimating heterogeneity was greater for small studies (median SE 0.39 dichotomous, 0.22 continuous) than for large studies (median SE 0.20 dichotomous, 0.13 continuous).

Conclusions:

• Heterogeneity is significantly larger in small studies compared to larger studies within meta-analyses.
• The substantial imprecision in estimating heterogeneity (τ) for small-study meta-analyses warrants caution.
• Sensitivity analyses are recommended when interpreting meta-analyses that include small studies with high heterogeneity.