Evidence quality in clinical guidelines: a comparison of two methods

Guidelines report the quality of the evidence used when formulating recommendations to ensure transparency and allow end-users to assess the estimates of effect that underlie the recommendation. The aim of this study was to investigate the differences in grading of evidence quality between the Scottish Intercollegiate Guideline Network (SIGN) model and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method.

The GRADE system was used to create evidence profiles for recommendations from a guideline that used the SIGN method to assess the quality of evidence. Scores were allocated to the quality assessment in its GRADE and SIGN forms, so that the difference between the two methods could be quantified.

The SIGN grading system led to quality ratings of the evidence of high and moderate but using the GRADE system, the majority of ratings were graded as low or very low. All of the recommendations were graded as higher quality by SIGN than by GRADE. The points' difference ranged from one to three points (maximum difference possible was four points). The most common reasons for downgrading evidence were "study limitations", "imprecision"' and "publication bias". "Indirectness" and "inconsistency" were rarely selected as reasons to downgrade the evidence.

The GRADE and SIGN methods produce varying estimates of the quality of evidence.