Related Experiment Videos
Single-reviewer assessments of certainty of evidence in network meta-analyses probably do not mislead
Carlos Zaror1, Róbinson Chávez-Concha2, Ariel Izcovich3
1Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Pediatric Dentistry and Orthodontics, Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile; Center for Research in Epidemiology, Economics and Oral Public Health (CIEESPO), Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile.
Background:
Systematic reviews with network meta-analyses (NMA) are the most comprehensive form of evidence synthesis for informing evidence-based practice recommendations. Rating the certainty of evidence (CoE) is a crucial step for drawing appropriate NMA's conclusions, but it can be time-consuming.
Objectives:
To evaluate the disagreements of assessments by a single reviewer (novice or expert) v consensus among two reviewers on CoE ratings of NMA used to support practice recommendations.
Methods:
We conducted a methodological case study using the grading of recommendations, assessment, development, and evaluations (GRADE) assessments for each comparison and outcome of interest from the COVID-19 living NMA, which informed the World Health Organization recommendations. Expert and novice reviewers assessed CoE independently and resolved disagreements by consensus. We defined an "important disagreement" in CoE ratings as any disagreement that would lead to an increase from high/moderate to low/very low or vice versa. We calculated the proportion of outcomes with important disagreements comparing the CoE rating from a single vs two reviewers. We performed a generalized linear mixed model analysis to compare the relative odds of disagreement between expert and novice reviewers.
Results:
Six experts and five novice reviewers rated 6652 comparisons. Compared with consensus ratings, novice reviewers exhibited higher disagreement rates across all GRADE domains, including assessments of direct, indirect, and network evidence. The disagreement rates for the overall CoE rating assessed by novice reviewers were higher than those of experts (10.6 vs 2.3%), but the important disagreement rate was similar (1.2% vs 1.0%). Novice reviewers were more likely to make important disagreements, but the association was not statistically significant (odds ratio, 1.3; 95% confidence interval 0.9-1.8).
Conclusion:
Novice reviewers disagreed with the consensus more often than expert reviewers when assessing the CoE from NMA. However, having a single reviewer was associated with relatively few disagreements likely to have meaningful implications for decision-makers.
Related Concept Videos
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Bias in Epidemiological Studies
Confounding in Epidemiological Studies
Strategies for Assessing and Addressing Confounding
Confounding can be addressed at both the design phase of a study and through analytical methods after data...
Confirmation Biases
Bias
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...