Controversy and debate on credibility ceilings. Paper 3: errors in the statistical justification for the "credibility...

We previously claimed that the credibility ceiling for meta-analyses is fundamentally flawed. We respond to Dr. Ioannidis' rebuttal of those claims.

We use statistical reasoning.

We agree with Dr. Ioannidis on some general points about the limitations of statistical sensitivity analyses. But critically, his response has entirely sidestepped responding to the crux of our argument, namely a direct mathematical demonstration that the method simply does not do what it was claimed to do. We reiterate that if our claim were false, it could be persuasively refuted if Dr. Ioannidis were to identify inaccuracies in our mathematical argument, which he has not done. Dr. Ioannidis had also dismissed as "absurd" the thought experiments we had used to illustrate the method's misleading conclusion; we explain why these examples still stand.

Given that the crux of our argument remains unaddressed, we continue to recommend against use of the credibility ceiling method. We are, however, sympathetic to what seem to be the underlying aims of the method, if not the execution. Developing principled methods to address those aims would be useful.