Increased risks for random errors are common in outcomes graded as high certainty of evidence

Area of Science:

• Evidence-based medicine
• Systematic reviews
• Meta-analysis

Background:

• The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system is widely used to assess the certainty of evidence (CoE).
• High CoE ratings indicate substantial confidence in the estimate of effect.
• However, random errors can still affect the reliability of outcomes, even with high CoE.

Purpose of the Study:

• To evaluate the prevalence of random errors in outcomes classified as high certainty of evidence (CoE).
• To identify factors associated with an increased risk of random errors in high CoE outcomes.

Main Methods:

• A random sample of 100 Cochrane reviews with dichotomous outcomes rated as high CoE was selected.
• Trial sequential analysis was performed to detect increased risks for random errors using standard thresholds for type I and type II errors.
• Multivariate logistic regression analyses identified predictors of increased risks for random errors.

Main Results:

• 38% of high CoE outcomes demonstrated an increased risk of random errors.
• Outcomes related to harms were more frequently affected (47%) than those assessing benefits (12%).
• Small absolute risk differences and a low number of events were significant predictors of increased random error risk.

Conclusions:

• A substantial proportion of high CoE outcomes are susceptible to random errors, potentially leading to false-positive or false-negative findings.
• Decision-makers must be aware of these risks when interpreting evidence rated as high CoE.
• Further methodological considerations are needed to mitigate random errors in evidence synthesis.