A scoping review of critical appraisal tools and user guides for systematic reviews with network meta-analysis:...

Systematic reviews (SRs) with network meta-analyses (NMAs) are increasingly used to inform guidelines, health technology assessments (HTAs), and policy decisions. Their methodological complexity, as well as the difficulty in assessing the exchangeability assumption and the large amount of results, makes appraisal more challenging than for SRs with pairwise NMAs. Numerous SR- and NMA-specific appraisal tools exist, but they vary in scope, intended users, and methodological guidance, and few have been validated.

To identify and describe appraisal instruments and interpretive guides for SRs and NMAs specifically, summarizing their characteristics, domain coverage, development methods, and measurement-property evaluations.

We conducted a methodological scoping review which included structured appraisal instruments or interpretive guides for SRs with or without NMA-specific domains, aimed at review authors, clinicians, guideline developers, or HTA assessors from published or gray literature in English. Searches (inception-August 2025) covered major databases, registries, organizational websites, and reference lists. Two reviewers independently screened records; data were extracted by one and checked by a second. We synthesized the findings narratively. First, we classified tools as either structured instruments or interpretive guides. Second, we grouped them according to their intended audience and scope. Third, we assessed available measurement-property data using relevant COnsensus-based Standards for the selection of health Measurement INstruments items.

Thirty-four articles described 22 instruments (11 NMA-specific, nine systematic reviews with meta-analysis-specific, 2 encompassing both systematic reviews with meta-analysis and NMA). NMA tools added domains such as network geometry, transitivity, and coherence, but guidance on transitivity evaluation, publication bias, and ranking was either limited or ineffective. Reviewer-focused tools were structured with explicit response options, whereas clinician-oriented guides posed appraisal questions with explanations but no prescribed response. Nine instruments reported measurement-property data, with validity and reliability varying widely.

This first comprehensive map of systematic reviews with meta-analysis and NMA appraisal resources highlights the need for clearer operational criteria, structured decision rules, and integrated rater training to improve reliability and align foundational SR domains with NMA-specific content.

NMA is a way to compare many treatments at once by combining results from multiple studies-even when some treatments have not been directly compared head-to-head. Because NMAs are complex, users need clear tools to judge whether an analysis is trustworthy. We reviewed and mapped 22 instruments published over the last 3 decades that are used to appraise or interpret SRs and NMAs. About half were designed specifically for NMAs; the rest were general SR tools that are applicable to NMAs. Most tools cover the basics of good reviews (clear question, fair search, bias assessment, and transparent synthesis). NMA-specific tools also address issues unique to networks, such as how the network is connected, whether indirect and direct evidence agree (consistency), and how to interpret treatment rankings. However, important gaps remain. Few tools give step-by-step checks for transitivity/consistency, network-level publication bias, or ranking uncertainty, and reported reliability between raters is inconsistent. Reporting checklists (eg, Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Network Meta-Analyses) specify what information should be reported but not how well it should be presented. Certainty frameworks (eg, Grading of Recommendations Assessment, Development, and Evaluation or Confidence in Network Meta-Analysis) outline how confidence in results is rated across domains such as inconsistency or imprecision, but they do not explain or standardize the different ways these domains are evaluated. What this means: guideline developers, HTA assessors, and clinicians should seek collaboration with statisticians experienced in NMA, and favor instruments with clear decision rules and user training. Better-tested, clearer tools will make NMA assessments more consistent and trustworthy.