The statistical properties of RCTs and a proposal for shrinkage.

Area of Science:

• Biostatistics
• Clinical Trial Design
• Medical Research Methodology

Background:

• Randomized controlled trials (RCTs) are fundamental in evidence-based medicine.
• Estimates from RCTs can be exaggerated, leading to the 'winner's curse' and replication failures.
• Statistical properties of RCT estimates, particularly the z-value and signal-to-noise ratio, are crucial for understanding bias.

Purpose of the Study:

• To abstract the concept of an RCT as a statistical triple $$ (parameter, estimate, standard error).
• To estimate the joint distribution of the z-value and signal-to-noise ratio from a large dataset of RCTs.
• To quantify the exaggeration ratio of effect estimates and propose a shrinkage method to correct for it.

Main Methods:

• Collected 23,551 pairs of (z-value, signal-to-noise ratio) from the Cochrane database.
• Estimated the distribution of achieved power and the exaggeration ratio.
• Developed and validated a shrinkage estimator for unbiased effect estimates.

Main Results:

• Median achieved power in RCTs was estimated to be only 13%.
• Estimates significant at the 5% level were found to overestimate the true effect by a factor of 1.7 on average.
• The proposed shrinkage estimator effectively reduces the exaggeration of effect estimates.

Conclusions:

• The winner's curse significantly impacts the reliability of RCT findings.
• Shrinkage estimation is a valuable method to correct for exaggerated effect sizes.
• Accurate effect size estimation is critical for reproducible and trustworthy scientific research.