Above the GRADE: Evaluation of Guidelines in Critical Care Medicine

We examined recommendations within critical care guidelines to describe the pairing patterns for strength of recommendation and quality of evidence. We further identified recommendations where the reported strength of recommendation was strong while the reported quality of evidence was not high/moderate and then assessed whether such pairings were within five paradigmatic situations offered by Grading of Recommendations Assessment, Development and Evaluation methodology to justify such pairings.

We identified all clinical critical care guidelines published online from 2011 to 2017 by the Society of Critical Care Medicine along with individual guidelines published by Surviving Sepsis Campaign, Kidney Disease Improving Global Outcomes, American Society for Parenteral and Enteral Nutrition, and the Infectious Disease Society of America/American Thoracic Society.

In all, 15 documents specifying 681 eligible recommendations demonstrated variation in strength of recommendation (strong n = 215 [31.6%], weak n = 345 [50.7%], none n = 121 [17.8%]) and in quality of evidence (high n = 41 [6.0%], moderate n = 151 [22.2%], low/very low n = 298 [43.8%], and Expert Consensus/none n = 191 [28.1%]). Strength of recommendation and quality of evidence were positively correlated (ρ = 0.66; p < 0.0001). Of 215 strong recommendations, 69 (32.1%) were discordantly paired with evidence other than high/moderate. Twenty-two of 69 (31.9%) involved Strong/Expert Consensus recommendations, a category discouraged by Grading of Recommendations Assessment, Development and Evaluation methodology. Forty-seven of 69 recommendations (68.1%) were comprised of Strong/Low or Strong/Very Low variation requiring justification within five paradigmatic scenarios. Among distribution in the five paradigmatic scenarios of Strong/Low and Strong/Very Low recommendations, the most common paradigmatic scenario was life threatening situation (n = 20/47; 42.6%). Four Strong/Low or Strong/Very Low recommendations (4/47; 8.5%) were outside Grading of Recommendations Assessment, Development and Evaluation methodology.

Among a large, diverse assembly of critical care guideline recommendations using Grading of Recommendations Assessment, Development and Evaluation methodology, the strength of evidence of a recommendation was generally associated with the quality of evidence. However, strong recommendations were not infrequently made in the absence of high/moderate quality of evidence. To improve clarity and uptake, future guideline statements may specify why such pairings were made, avoid such pairings when outside of Grading of Recommendations Assessment, Development and Evaluation criteria, and consider separate language for Expert Consensus recommendations (good practice statements).