A threshold analysis assessed the credibility of conclusions from network meta-analysis

Area of Science:

• Health research methodology
• Evidence-based medicine
• Biostatistics

Background:

• Network meta-analysis (NMA) is widely used to synthesize evidence from multiple studies.
• Assessing the reliability of treatment recommendations from NMAs is crucial for clinical decision-making.
• Existing methods like Grading of Recommendations Assessment, Development and Evaluation (GRADE) have limitations in evaluating NMA reliability.

Purpose of the Study:

• To develop and assess a novel threshold analysis method for evaluating the reliability of NMA treatment recommendations.
• To determine how much bias in the evidence base would be required to alter a treatment recommendation.
• To compare the results of threshold analysis with existing GRADE assessments for NMAs.

Main Methods:

• A structured series of threshold analyses were applied to pairwise contrasts within two NMAs (weight loss and osteoporosis).
• The analyses quantified the magnitude of bias needed to change treatment decisions.
• The Grading of Recommendations Assessment, Development and Evaluation (GRADE) system was also implemented for comparison.

Main Results:

• In the weight-loss NMA, plausible bias in 3 out of 10 estimates could have changed the treatment recommendation.
• For the osteoporosis NMA, plausible bias in 6 out of 16 estimates could alter the recommendation.
• No correlation was found between the original GRADE assessments and the impact of plausible bias on treatment recommendations.

Conclusions:

• Individual reliability judgments of NMA contrasts do not sufficiently inform decision-makers about recommendation robustness.
• Threshold analysis provides a more robust method for assessing whether NMA treatment recommendations are reliable against potential bias.
• This method enhances the trustworthiness of evidence synthesis in clinical practice and health policy.