Use of thresholds for rating certainty of evidence with the GRADE framework: a systematic survey of systematic...

Current guidance regarding applying the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework mandates establishing thresholds to assess certainty of evidence. These thresholds include the null, the minimally important difference (MID), and moderate and large effect thresholds. GRADE experts offer a rationale for use of each particular threshold. However, GRADE methodologists continue to debate the optimal choice of threshold, particularly whether to use the null threshold. We aimed to describe how systematic review authors used thresholds and in doing so to provide insight into the perceived utility of the available thresholds and to inform ongoing debate regarding the optimal choice of threshold.

We conducted a systematic survey sampling the 200 most recently published Cochrane and 200 non-Cochrane systematic reviews that used GRADE, which we identified from the Cochrane Database of Systematic Reviews (2018-July 2024) and MEDLINE (2018-December 2024). We documented whether review authors used thresholds and the thresholds they chose.

Among the sampled reviews, 118 of 200 (59%) Cochrane and 63 of 200 (31.5%) non-Cochrane reviews used thresholds. Among the reviews with an identifiable threshold type, 55 of 112 (49.1%) Cochrane and 26 of 55 (47.3%) non-Cochrane reviews used the null, whereas 54 of 112 (48.2%) Cochrane and 24 of 55 (43.6%) non-Cochrane reviews used the MID. Few reviews used the MID, moderate, and large effect thresholds together to form ranges of effects (two of 112 [1.8%] Cochrane; three of 55 [5.5%] non-Cochrane).

Use of thresholds in systematic reviews using the GRADE framework remains suboptimal, particularly in non-Cochrane reviews. For systematic reviews that do use thresholds, reviewers choose the null and the MID equally and very seldom use multiple thresholds, suggesting the perceived utility of the available options.

Systematic reviews of interventions provide evidence to support health decisions by summarizing findings from available studies. Review authors assess their confidence in the evidence using a system called GRADE (Grading of Recommendations Assessment, Development and Evaluation). A key part of GRADE is setting thresholds that guide certainty ratings. A common threshold is the null, which addresses the question of whether or not there is any effect at all. Another is the minimally important difference (MID), which considers whether the effect is important to patients. We analyzed 400 systematic reviews to examine how their authors used these thresholds. Over half of the reviews did not use thresholds. Among those that did, approximately equal numbers used the null threshold and the MID. Our findings establish that review authors who use thresholds choose the null and the MID equally often when rating GRADE certainty.