Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

Area of Science:

• Medical Research Integrity
• Biostatistics
• Publication Bias

Background:

• Concerns about the trustworthiness of scientific literature have been amplified by claims that most published results are false.
• Previous empirical investigations into the rate of false-positive results have not definitively resolved this issue.

Purpose of the Study:

• To propose and apply a novel method for estimating the false positive risk in published research.
• To empirically evaluate the false positive risk in randomized clinical trials published in leading medical journals.

Main Methods:

• Development of a new statistical approach to quantify false positive risk.
• Application of the method to a dataset of clinical trial results from top-tier medical journals.
• Analysis of the impact of traditional significance thresholds (e.g., alpha = 0.05) on false positive rates.

Main Results:

• Contrary to widespread claims, the study found that the traditional significance level of alpha = 0.05 yields a false positive risk of 13%.
• Adjusting the significance threshold to alpha = 0.01 substantially reduces the false positive risk to below 5%.
• The study identified clear evidence of publication bias, which contributes to inflated estimates of effect sizes.

Conclusions:

• The findings challenge the notion that most published research results are false.
• A revised significance threshold (alpha = 0.01) offers a more reliable standard for reducing false positives in medical research.
• Addressing publication bias is crucial for accurate effect size estimation and enhancing the overall trustworthiness of medical research findings.