Hartung-Knapp method is not always conservative compared with fixed-effect meta-analysis

A widely used method in classic random-effects meta-analysis is the DerSimonian-Laird method. An alternative meta-analytical approach is the Hartung-Knapp method. This article reports results of an empirical comparison and a simulation study of these two methods and presents corresponding analytical results. For the empirical evaluation, we took 157 meta-analyses with binary outcomes, analysed each one using both methods and performed a comparison of the results based on treatment estimates, standard errors and associated P-values. In several simulation scenarios, we systematically evaluated coverage probabilities and confidence interval lengths. Generally, results are more conservative with the Hartung-Knapp method, giving wider confidence intervals and larger P-values for the overall treatment effect. However, in some meta-analyses with very homogeneous individual treatment results, the Hartung-Knapp method yields narrower confidence intervals and smaller P-values than the classic random-effects method, which in this situation, actually reduces to a fixed-effect meta-analysis. Therefore, it is recommended to conduct a sensitivity analysis based on the fixed-effect model instead of solely relying on the result of the Hartung-Knapp random-effects meta-analysis. Copyright © 2016 John Wiley & Sons, Ltd.