Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and...

Area of Science:

• Biostatistics
• Evidence Synthesis
• Critical Care Medicine

Background:

• Combining randomized controlled trials (RCT) and non-randomized studies (NRS) in meta-analysis is debated due to potential bias from NRS.
• Assessing treatment effects requires robust methods that can account for inherent differences between study designs.

Purpose of the Study:

• To compare a Bayesian meta-analytic approach with a frequentist approach for pooling RCT and NRS data.
• To evaluate how each method handles potential bias introduced by non-randomized studies.

Main Methods:

• A systematic search identified binary outcome meta-analyses in Critical-Care combining RCT and NRS.
• Bayesian pooled treatment-effects and credible intervals were estimated using a half-Cauchy prior, with Bayes-factors (BF) determining model preference.
• Frequentist pooled treatment-effects and confidence intervals were re-estimated using the DerSimonian-Laird (DSL) random effects model.

Main Results:

• Out of 50 identified meta-analyses, 44 reported pooled estimates.
• Frequentist confidence intervals excluded the null in 86% of cases, while Bayesian credible intervals embraced the null in 23 instances where frequentist intervals did not.
• Bayes-factors supported a pooled model in 27 meta-analyses, indicating preference for integrating RCT and NRS data.

Conclusions:

• An integrated Bayesian approach provides support for including non-randomized studies in pooled meta-analytic treatment-effect models.
• Caution is advised when reporting frequentist pooled treatment effects that combine randomized controlled trials and non-randomized studies without careful consideration of potential bias.