Related Experiment Videos
SINGLE REVIEWER ASSESSMENTS OF CERTAINTY OF EVIDENCE IN NETWORK META-ANALYSES PROBABLY DO NOT MISLEAD
Carlos Zaror1, Róbinson Chávez-Concha2, Ariel Izcovich3
1Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada; Department of Pediatric Dentistry and Orthodontics, Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile; Center for Research in Epidemiology, Economics and Oral Public Health (CIEESPO), Faculty of Dentistry, Universidad de La Frontera, Temuco, Chile.
Journal of Clinical Epidemiology
|August 4, 2026
Summary
Assessing certainty of evidence (CoE) in network meta-analyses (NMA) can be time-consuming. While novice reviewers disagreed more often than experts, single reviewer assessments had few disagreements impacting decisions.
Area of Science:
- Evidence synthesis
- Health research methodology
Background:
- Network meta-analyses (NMA) are crucial for evidence-based recommendations.
- Rating certainty of evidence (CoE) is vital but time-intensive.
Purpose of the Study:
- To compare CoE rating disagreements between single reviewers (novice/expert) and consensus of two reviewers.
- To evaluate the impact of reviewer experience on CoE assessment in NMAs.
Main Methods:
- Methodological case study using GRADE assessments from the COVID-19 Living NMA.
- Independent CoE ratings by novice and expert reviewers, with consensus for disagreements.
- Analysis of "important disagreements" and comparison of disagreement rates using generalized linear mixed models.
Main Results:
- Novice reviewers had higher disagreement rates than experts across all GRADE domains.
- Overall disagreement rates for novices were higher (10.6%) than experts (2.3%).
- Important disagreement rates were similar (1.2% vs 1.0%), with no statistically significant difference in novice reviewers' likelihood of important disagreements.
Conclusions:
- Novice reviewers disagreed more with consensus than experts in NMA CoE ratings.
- Single reviewer assessments showed few disagreements with significant decision-making implications.
Related Concept Videos
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
01:29Bias in Epidemiological Studies
Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:
01:27Confounding in Epidemiological Studies
Confounding in statistical epidemiology represents a pivotal challenge, referring to the distortion in the perceived relationship between an exposure and an outcome due to the presence of a third variable, known as a confounder. This variable is associated with both the exposure and the outcome but is not a direct link in their causal chain. Its presence can lead to erroneous interpretations of the exposure's effect, either exaggerating or underestimating the true association. This phenomenon...
01:25Strategies for Assessing and Addressing Confounding
Confounding is a critical issue in epidemiological studies, often leading to misleading conclusions about associations between exposures and outcomes. It occurs when the relationship between the exposure and the outcome is mixed with the effects of other factors that influence the outcome. Given that, addressing confounding is of high importance for drawing accurate inferences in research.
Confounding can be addressed at both the design phase of a study and through analytical methods after data...
Confounding can be addressed at both the design phase of a study and through analytical methods after data...
01:31Confirmation Biases
The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
01:22Bias
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...