Controversy and debate on credibility ceilings. Paper 1: Fundamental problems with the "credibility ceiling" method...

The "credibility ceiling" method was proposed to conduct sensitivity analysis for unmeasured confounding and other forms of bias in meta-analyses and has been used in umbrella reviews to grade evidence strength. However, we explain that the method has fundamental statistical flaws.

We use statistical reasoning to assess the method's validity, providing intuition for our findings by presenting simple applied examples in which the method yields clearly incorrect conclusions.

The credibility ceiling is not a valid bias correction, as we show mathematically and illustrate using examples in which, for example, the method incorrectly "adjusts" the meta-analytic point estimate in the wrong direction. Although the originators describe the method as limiting the credibility of any given observational study to a fixed ceiling, we show why this interpretation in fact bears little relation to what the method actually does.

Given the fundamental problems with the credibility ceiling method and its demonstrated potential for misleading conclusions, we recommend against its use.