Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the...

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Meta-analyses often combine studies with varying designs, leading to a lack of a common true effect size.
• Estimates of between-study variance are crucial for interpreting meta-analytic results but are frequently imprecise.
• Confidence intervals for between-study variance aid interpretation, but their accuracy depends on underlying model assumptions.

Purpose of the Study:

• To evaluate the accuracy of confidence intervals for between-study variance in meta-analysis.
• To assess the performance of the Q-profile and generalized Q-statistic methods under practical meta-analytic conditions.

Main Methods:

• Utilized two Monte-Carlo simulation studies.
• Employed odds ratio as the effect size measure.
• Investigated scenarios representative of real-world meta-analyses, including small primary study sample sizes.

Main Results:

• Coverage probabilities for both Q-profile and generalized Q-statistic methods were often substantially below nominal rates.
• These inaccuracies were linked to violations of random-effects model assumptions, specifically non-normal sampling distributions and unknown sampling variances.
• Performance degradation was particularly pronounced when primary study sample sizes were small.

Conclusions:

• The commonly used Q-profile and generalized Q-statistic methods may yield unreliable confidence intervals for between-study variance.
• Violations of random-effects model assumptions, exacerbated by small sample sizes, are primary drivers of inaccurate interval coverage.
• Researchers should exercise caution when interpreting confidence intervals for between-study variance in meta-analyses with heterogeneous study designs or small sample sizes.