GRADE guidance 36: updates to GRADE's approach to addressing inconsistency

To update previous Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance by addressing inconsistencies and interpreting subgroup analyses.

Using an iterative process, we consulted with members of the GRADE working group through multiple rounds of written feedback and discussions at GRADE working group meetings.

The guidance complements previous guidance with clarification in two areas: (1) assessing inconsistency and (2) assessing the credibility of possible effect modifiers that might explain inconsistency. Specifically, the guidance clarifies that inconsistency refers to variability in results, not in study characteristics; that inconsistency assessment for binary outcomes requires consideration of both relative and absolute effects; how to decide between narrower and broader questions in systematic reviews and guidelines; that, with the same evidence, ratings of inconsistency may differ depending on the target of certainty rating; and how GRADE inconsistency ratings relate to a statistical measure of inconsistency I² depending on the context in which one views results. The second part of the guidance illustrates, based on a worked example, the use of the instrument to assess the credibility of effect modification analyses. The guidance explains the stepwise process of moving from a subgroup analysis to assessing the credibility of effect modification and, if found credible, to subgroup-specific effect estimates and GRADE certainty ratings.

This updated guidance addresses specific conceptual and practical issues that systematic review authors frequently face when considering the degree of inconsistency in estimates of treatment effects across studies.