Disentangling the GRADE Domains of Inconsistency and Imprecision: An Approach Based on a Meta-Research Study

The assessment of the certainty of evidence using GRADE requires evaluating inconsistency and imprecision. These domains are difficult to judge in a fully independent way. We performed a meta-research study to propose an approach to disentangle inconsistency and imprecision based on decision thresholds.

We evaluated systematic reviews published in the Cochrane Database of Systematic Reviews. We included all meta-analyses published in these reviews (i) with at least five primary studies, (ii) which did not correspond to subgroup analyses, and (iii) which had provided their results as continuous outcomes or as dichotomous outcomes with previously established decision thresholds. We recalculated each included meta-analysis using the fixed- and the random-effects models, retrieving metrics potentially expressing inconsistency or imprecision. This includes inconsistency indices, variables related to the confidence intervals (CI) of the meta-analytical estimate and prediction intervals. Based on these variables, we performed factor analysis, hierarchical cluster analysis, and decision tree analysis.

We assessed 8703 meta-analyses. The factor analysis resulted in the identification of three factors: (i) one factor associated with inconsistency metrics, (ii) one factor associated with imprecision metrics, and (iii) one factor of metrics not clearly related to inconsistency or imprecision. Taken together, the analysis suggests an approach based on the number of decision thresholds crossed by the CI of the fixed- and random-effects meta-analytical estimates: the number of thresholds crossed by the CI of the fixed-effect model informs about imprecision, while the difference in the number of thresholds crossed by the CIs of the random- versus the fixed-effect models reflects inconsistency. Prediction intervals and inconsistency indices, such as the I ², can help detect cases in which inconsistency may be underestimated.

Our results may help those assessing the certainty in evidence to independently rate inconsistency and imprecision.