Meta-analyses with binary outcomes: how many studies need to be omitted to detect a publication bias?

Meta- and pooled analyses are increasingly applied to aggregate the results of a number of studies, especially in health sciences. A typical difficulty is the presence of a publication bias. Usually Egger's regression test and funnel plots are applied to detect such a publication bias. A simulation study was conducted to investigate the quantity of null and negative results required to be omitted to detect a publication bias. In particular, the performance of Egger's test and funnel plots was considered in two scenarios with binary outcomes and expected odds ratios (OR) of 1 and 2, respectively. For both scenarios Egger's test detected only a small fraction of publication biases if few studies were deleted, corresponding to the results of a random deletion. Moreover, if a true null effect is present Egger's test is quite unlikely to detect a publication bias even if a considerable proportion of the null results are missing. Generally, the detection of a publication bias using Egger's test is only likely if both the "true" effect and the bias are large enough. Visual inspection of the funnel plots resulted in a higher fraction of detected publication biases in cases where a bias was present and in cases where studies were randomly deleted, revealing the arbitrariness of this method. Evidence indicates that standard methods for detection of a publication bias do not necessarily detect such a bias; thus, additional tests for publication bias need to be applied.