Meta-research: The art of getting it wrong

Meta-analysis has major strengths, but sometimes it can often lead to wrong and misleading answers. In this SRSM presidential address, I discuss some case studies that exemplify these problems, including examples from meta-analyses of both clinical trials and observational associations. I also discuss issues of effect size estimation, bias (in particular significance-chasing biases), and credibility in meta-research. I examine the factors that affect the credibility of meta-analyses, including magnitude of effects, multiplicity of analyses, scale of data, flexibility of analyses, reporting, and conflicts of interest. Under the current circumstances, a survey of expert meta-analysts attending the SRSM meeting showed that most of them believe that the true effect is practically equally likely to lie within the 95% confidence interval of a meta-analysis or outside of it. Finally, I address the placement of meta-analysis in the wider current research agenda and make a plea for adoption of more prospective meta-designs. In many/most/all fields, all primary original research may be designed, executed, and interpreted as a prospective meta-analysis. Copyright © 2011 John Wiley & Sons, Ltd.