GRADE guidelines system is reproducible when instructions are clearly operationalized even among the guidelines panel...

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) is a widely used methodology for the development of clinical practice guideline. Although its reproducibility is good for evaluating the quality of evidence, it has not been tested in context of developing recommendations. The objective of this study was to assess the reproducibility of all GRADE factors that determine the direction and strength of a recommendation among the guideline panel members with limited exposure to GRADE methodology.

The study was conducted as part of the clinical practice guideline development process of American Association of Blood Banking for the use of prophylactic vs. therapeutic platelet transfusion in patients with thrombocytopenia. The results from the systematic review and meta-analysis for each question were summarized as a GRADE evidence profile. Interrater agreement for all GRADE factors and strength of recommendations was summarized using a weighted kappa statistic with 95% confidence intervals (CI).

Eighteen members of the panel participated in the deliberation of making recommendations and completed the online questionnaire. They were given two 1-hour lectures about GRADE. The agreement for all domains was better than chance. The interrater agreement for the domain of quality of evidence was good (kappa value: 0.68; 95% CIs: 0.54, 0.84), and fair for balance of benefit and harms (kappa value: 0.4; 95% CIs: 0.25, 0.57) and use of resources (kappa value: 0.28: 95% CIs: 0.12, 0.42). The interrater agreement was moderate for the GRADE domain of patients' values and preferences (kappa value: 0.44; 95% CI: 0.31, 0.56). The interrater agreement for making a for/against recommendation was good (kappa value: 0.74; 95% CIs: 0.33, 0.91) and fair for strong/weak recommendation (kappa value: 0.39; 95% CIs: 0.18, 0.68).

Although not all elements of GRADE system had good agreement, the interrater agreement for assessing the quality of evidence and issuing a recommendation of for vs. against among panel members who had limited exposure to GRADE methodology was good. This is probably because GRADE has operationalized these two areas in more detail than other domains. Further operationalization of all GRADE domains such as with the GRADE evidence to decision frameworks would likely improve its reproducibility.