Consequences of sequential sampling for meta-analysis

Sequential stopping rules allow hypotheses to be tested using smaller sample sizes than are possible under conventional methods, while controlling the Type I and II error rates. However, the consequences of using such procedures when combining studies in a meta-analysis have rarely been discussed. For a primary study to be included in a meta-analysis, it must provide an estimate of the effect size, and it must be possible to calculate the variance of this estimate, which is used for weighting the study. It is therefore crucial to know whether the use of sequential stopping rules introduces any bias in the estimate of the effect size and/or modifies the variance of the estimate. In the present research, both aspects were studied for the CLAST rule, as applied to testing the difference between two means from paired samples, in a variety of scenarios of sample size and population effect size. The results show that although the bias is small, but still larger than that for the fixed-sample rule, the variance of the estimate is much higher with the CLAST sequential stopping rule. The implications of these results for the incorporation of such studies into meta-analyses are discussed. It is recommended to incorporate such studies into meta-analyses by taking only the information conveyed in the initial sample. The authors of primary studies employing sequential rules should report that information when publishing their results.