Evaluating cumulated research. I: The inadequacy of traditional methods

Virtually all reviews of cumulated studies rely on statistical significance as a criterion for evaluating the reproducibility of the phenomenon under review. Despite its nearly universal application, that criterion is entirely inadequate: Its application is very likely to lead a reviewer to conclude that a phenomenon does not discriminate patients from controls when, in fact, it does do so. The reviewer is, paradoxically, more likely to draw this incorrect conclusion as more studies become available for review. It can lead a reviewer to conclude that one phenomenon is more discriminating than another when the opposite is actually true. Fortunately, procedures that do not distort the review process are available; some of these are briefly discussed.