Evaluating evidential strength of statistically nonsignificant meta-analyses with likelihood ratios: a...

This study aimed to determine the proportion of statistically nonsignificant meta-analyses that provide evidence favoring the null hypothesis of no difference, evidence favoring the alternative hypothesis, or inconclusive findings using likelihood ratios.

In this meta-epidemiological study, Cochrane systematic reviews published in 2025 were screened. Eligible meta-analyses pooled data from two or more primary studies, reported a pooled effect estimate with a 95% CI including the value of no difference, and had a P value between 0.05 and 0.20. Meta-analyses using Hartung-Knapp/Sidik-Jonkman were excluded. In the primary analysis, likelihood ratios compared the null hypothesis against the observed pooled effect. In the secondary analysis, likelihood ratios were calculated against prespecified small, medium, and large effect thresholds; meta-analyses reporting weighted mean differences were excluded as no agreed-upon thresholds exist for these outcomes.

A total of 936 meta-analyses from 172 Cochrane reviews were eligible. In the primary analysis, 693 (74%) provided weak evidence favoring the alternative hypothesis, and 243 (26%) were inconclusive; none favored the null hypothesis. All meta-analyses with P values between .05 and .10 showed weak evidence favoring the alternative, while those with P values between .11 and .20 were split between weak evidence favoring the alternative (55%) and inconclusive (45%) findings. In the secondary analysis of 732 meta-analyses, the proportion supporting the null hypothesis increased from 20% for small to 55% for medium to 75% for large effect thresholds, while inconclusive findings declined correspondingly.

Most nonsignificant meta-analyses provide weak evidence favoring the observed treatment effect rather than supporting no difference, but evidence increasingly favors no difference as prespecified effect thresholds increase in magnitude. Likelihood ratios enable guideline developers and clinicians to distinguish evidence of no difference from inconclusive findings and support more informed clinical recommendations and better-targeted research prioritization.

When research combines results from multiple studies (called meta-analyses) and finds no statistically significant difference between treatments, researchers often interpret this to mean "there is no difference between the treatments." However, this can be a mistake. Sometimes the data truly show the treatments work the same, but other times the data are simply unclear, and we just do not have enough information to tell either way. We used a statistical method called likelihood ratios to determine whether meta-analyses supported "no difference" or whether the results were simply unclear or favored a difference between the treatments. By examining 936 nonsignificant meta-analyses, we found that most results (74%) provided weak evidence that there is a difference between treatments, not that treatments work the same. However, when we compared results to what would count as clinically important differences, the picture changed. For small important differences, most results were unclear, while for large important differences, 75% supported "no difference." This matters because misinterpreting unclear results as showing "no difference" can affect medical guidelines and patient care. It can also lead to wasted research by repeating studies when evidence already supports no difference or by stopping research when results are simply unclear and more data are needed. When researchers analyze combined results from studies, they should specify in advance what size of treatment difference would matter to patients, then use likelihood ratios to clearly state whether their results support no difference, are unclear, or suggest a treatment effect. This will help clinicians make better treatment decisions and researchers prioritize which questions still need more study.