Rethinking methodological appraisal in overviews of systematic reviews: a participatory validation framework

Overviews of systematic reviews, or umbrella reviews, synthesize evidence across related research questions and inform clinical practice, policy, and research prioritization. A core component of overviews is the methodological appraisal of included systematic reviews, which guides interpretation, weighting of evidence, and confidence in conclusions. Although structured tools such as A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2) and the Risk of Bias in Systematic Reviews (ROBIS) are commonly applied, these assessments remain interpretive, requiring judgment about reporting adequacy, methodological decisions, and the distinction between conduct and reporting limitations. Evidence shows substantial variability in ratings across research teams, even when using standardized instruments, and systematic review authorship or professional allegiance can influence appraisals, potentially affecting conclusions and comparability across overviews.

To address these challenges, this commentary proposes a participatory validation framework in which overview authors conduct independent methodological appraisals and subsequently invite authors of included systematic reviews to comment on and reassess these judgments. Responses are compared with the original assessments, and areas of agreement and disagreement are transparently documented without requiring consensus. The framework is intended to enhance transparency and interpretability by making concordance and discordance visible rather than to establish a single definitive rating or resolve all disagreement. The framework is feasible within existing workflows, with minimal disruption beyond sharing assessments and documenting responses. Ethical considerations, such as potential self-serving bias, are mitigated through independent initial appraisal, anonymous reassessment, and transparent reporting. While adoption may be limited by selective participation and administrative burden, the framework represents a pragmatic, exploratory strategy to improve transparency, credibility, and interpretability of methodological appraisal in overviews, providing richer, more informative guidance for authors, guideline developers, policymakers, and the broader evidence synthesis community.