How to quantify between-study heterogeneity in single-arm evidence synthesis?-It depends!

Area of Science:

• Biostatistics
• Meta-analysis methodology
• Evidence synthesis

Background:

• Random-effects meta-analysis models require estimating heterogeneity variance (${\tau }^2$).
• No consensus exists on optimal heterogeneity variance estimators, particularly for single-arm observational studies facing unique challenges.
• Existing literature lacks systematic comparisons of heterogeneity variance estimators in these specific contexts.

Purpose of the Study:

• To investigate the performance of different heterogeneity variance estimators in typical single-arm meta-analysis scenarios.
• To compare the advantages of various estimators through simulations and an empirical pediatric application.
• To assess current practices and reporting quality of heterogeneity variance estimators in high-ranked journals.

Main Methods:

• Compared seven heterogeneity variance estimators for random-effects meta-analysis based on methodological diversity.
• Conducted simulation studies for continuous and binary outcomes in single-arm meta-analysis settings.
• Performed a non-systematic literature review to evaluate current usage and reporting quality of estimators in journals.

Main Results:

• All evaluated heterogeneity estimators demonstrated imprecision and often failed to capture true heterogeneity, especially with few studies or rare binary events.
• Many estimators produced zero heterogeneity estimates even when heterogeneity was present.
• Overall effect estimates were robust across estimators, but prediction intervals varied significantly, highlighting estimator-dependent uncertainty.

Conclusions:

• Substantial differences in heterogeneity variance estimates exist between estimators, yet their selection in single-arm meta-analysis receives insufficient attention.
• Increased awareness and practical consideration of different heterogeneity variance estimators and their properties are needed.
• Recommends evaluating a range of plausible estimators via sensitivity analysis before finalizing meta-analysis conclusions.