Random error in cardiovascular meta-analyses: how common are false positive and false negative results?

Area of Science:

• Medical research methodology
• Evidence-based medicine
• Biostatistics

Background:

• Cochrane reviews are considered the gold standard for meta-analyses due to rigorous methods for minimizing systematic error.
• However, the impact of random error on meta-analysis conclusions is less frequently addressed.
• Spurious conclusions in meta-analyses can arise from both systematic and random errors.

Purpose of the Study:

• To assess the frequency of false positive and false negative findings in Cochrane meta-analyses.
• To evaluate the adequacy of sample sizes in detecting a relative risk reduction (RRR) of at least 25% using trial sequential analysis.
• To determine if robust reviews, such as those from the Cochrane Heart Group, are susceptible to random error leading to incorrect conclusions.

Main Methods:

• Examined Cochrane reviews published by the Cochrane Heart Group in 2012 with at least five randomized trials.
• Applied trial sequential analysis (TSA) to statistically significant meta-analyses to verify true positives against O'Brien-Fleming boundaries.
• Classified non-significant meta-analyses as true negatives if sample size was sufficient to reject a 25% RRR.

Main Results:

• 41% of meta-analyses reported statistically significant results, with 83% being true positives.
• 36% of non-significant meta-analyses were true negatives.
• 45% of all meta-analyses lacked sufficient sample size to detect or rule out a 25% RRR, with 55% of those authors deemed conclusive having insufficient data.

Conclusions:

• False positive and false negative meta-analyses are prevalent and often unrecognized, even within high-quality Cochrane reviews.
• The study highlights the potential for random error to cause misleading conclusions.
• Incorporating trial sequential analysis is recommended for both meta-analysts and readers to improve the interpretation of meta-analysis results.