Systematic evaluation and comparison of statistical tests for publication bias

Area of Science:

• Biostatistics
• Medical Research Methodology
• Systematic Reviews

Background:

• Publication bias poses a significant threat to the validity of meta-analyses.
• Statistical methods are crucial for identifying publication bias.

Purpose of the Study:

• To compare the statistical and discriminatory powers of Begg's, Egger's, and Macaskill's methods for detecting publication bias.
• To evaluate the impact of varying p-value cutoffs on the performance of these bias detection methods.

Main Methods:

• Utilized 130 Cochrane reviews with binary endpoints and at least 10 studies.
• Employed funnel plots with observer agreement as a reference standard.
• Assessed sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve by adjusting p-value thresholds.

Main Results:

• Egger's method demonstrated higher sensitivity (0.93) than Begg's (0.86) and Macaskill's (0.43) at a 0.1 false positive rate.
• The area under the ROC curve for Egger's method (0.955) and Begg's method (0.913) were comparable and superior to Macaskill's (0.719).
• Increasing p-value cutoffs improved test sensitivity with minimal impact on specificity.

Conclusions:

• Egger's and Begg's methods exhibit superior statistical and discriminatory power for detecting publication bias compared to Macaskill's method.
• Adjusting the p-value cutoff can enhance the power of these bias detection techniques without substantially increasing the false positive rate.