Why most discovered true associations are inflated

Area of Science:

• Statistical genetics
• Biostatistics
• Epidemiology

Background:

• Newly discovered associations in scientific research, particularly in fields like genomics and clinical trials, frequently exhibit effect sizes that are larger than the true underlying effects.
• This inflation is a well-documented phenomenon across various research domains.

Purpose of the Study:

• To elucidate the primary reasons behind the inflation of effect sizes in newly discovered true associations.
• To discuss potential strategies for addressing and mitigating this observed inflation.

Main Methods:

• Theoretical considerations regarding statistical significance thresholds and study power.
• Analysis of flexible statistical analyses and selective reporting practices.
• Examination of interpretation biases influenced by conflicts of interest.

Main Results:

• Underpowered studies claiming discovery based on statistical significance thresholds lead to expected effect inflation.
• Flexible data analysis combined with selective reporting can significantly inflate observed effects (high vibration ratio).
• Conflicts of interest during interpretation can also contribute to effect inflation, though deflation can occur in specific scenarios like late discoveries in overpowered studies or due to measurement error.

Conclusions:

• Effect inflation in newly discovered associations is a multifaceted problem driven by statistical, analytical, and interpretative factors.
• Recommendations include cautious interpretation of initial effect sizes, employing analytical methods for inflation correction, prioritizing large discovery studies, enforcing strict analytical protocols, ensuring transparent reporting, emphasizing replication, and maintaining unbiased interpretation.