Accumulating evidence from independent studies: what we can win and what we can lose

When asking 'what is known' about a drug or therapy or program at any time, both researchers and practitioners often confront more than a single study. Facing a variety of findings, where conflicts may outweigh agreement, how can a reviewer constructively approach the task? In this discussion, I will outline some questions that can only be answered by examining a group of independent studies. I will also discuss some pitfalls that sometimes swamp the benefits we can gain from synthesis. Most of these pitfalls are avoidable if anticipated early in a review. The benefits of a quantitative review include information about how to match a treatment with the most promising recipients; increasing the statistical power to detect a significant new treatment; telling us when 'contextual effects' are important; helping us to assess the stability and robustness of treatment effectiveness; and informing us when research finds are especially sensitive to investigators' research design. The pitfalls include aggregating data from studies on different populations; aggregating when there is more than one underlying measure of central tendency; and emphasizing an average outcome when partitioning variance gives far more useful information.