Level of Evidence and Strength of Recommendations in US Medical Society Clinical Practice Guidelines, 2019-2023: A...

Clinicians rely on medical societies' clinical practice guidelines to inform evidence-based patient care. Concordance between these guidelines' strength of recommendation (SOR) ratings and corresponding level of evidence (LOE) ratings is unclear.

Evaluate concordance between SOR and LOE ratings for guideline recommendations.

Cross-sectional analysis.

Recommendations from 309 clinical practice guideline documents issued by 23 US-based medical societies from 2019 to 2023.

Recommendation-level variables: SOR and LOE ratings. Society-level variables: methodology for assigning SOR and LOE ratings and society's specialty type. SOR-LOE ratings were concordant if the LOE rating was at least as strong as the SOR rating, with concordance quantified using modified Cohen's kappa [κ].

Among 7582 recommendations, society guideline writing committees rated SOR as strong for 2303 (30%) recommendations, moderate for 1351 (18%), weak for 2028 (27%), and 1900 recommendations (25%) had no strength assigned. Regarding LOE ratings, 768 (10%) recommendations had strong evidence, 1676 (22%) had moderate evidence, 2421 (32%) had weak or very weak evidence, and 2717 (36%) had evidence lacking an LOE rating. Overall, 5088 (67.1%) recommendations had SOR-LOE rating concordance, and overall SOR-LOE rating concordance was moderate (κ = 0.67; 95% confidence interval (CI) 0.66-0.68). Strong recommendations, however, had minimal SOR-LOE concordance (κ = 0.27; 95% CI 0.25-0.28; p < 0.001), with 1690 (73%) having LOE ratings weaker than strong. Recommendations based on a Modified Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework (κ = 0.33; 95% CI 0.30-0.37, p < 0.001), and issued by obstetrics/gynecology (κ = 0.46; 95% CI 0.40-0.51, p < 0.001) or neurology/psychiatry (κ = 0.22; 95% CI 0.18-0.26, p < 0.001) societies had significantly lower rating concordance.

Among recently issued CPG recommendations, overall SOR-LOE rating concordance was moderate, although recommendations with strong SOR ratings had minimal rating concordance. CPG authors could provide clearer rationales for strong recommendations when the corresponding evidence does not reflect commensurate certainty.