You Cannot Step Into the Same River Twice: When Power Analyses Are Optimistic.

Area of Science:

• Psychological research methodology
• Statistical inference
• Quantitative psychology

Background:

• Statistical power is crucial for detecting effects but relies on effect size estimates.
• Effect sizes in psychology are highly variable due to diverse study designs and contexts.
• Standard power formulas neglect this between-study variability, leading to inflated power estimates.

Purpose of the Study:

• To address the issue of optimistic power assessments in psychological research.
• To develop and propose revised power calculation formulas that incorporate between-study effect size variation.
• To provide practical tools and recommendations for researchers to determine accurate sample sizes for studies and replications.

Main Methods:

• Development of novel statistical power formulas accounting for systematic variation in effect sizes across studies.
• Illustration of the impact of between-study variation using hypothetical examples and empirical data from psychology.
• Introduction of a user-friendly online tool to implement the proposed sample size setting approach.

Main Results:

• Standard power calculations overestimate statistical power when between-study effect size variation is present.
• The proposed conservative formulas yield more realistic power assessments, reducing the likelihood of underpowered studies.
• The developed approach is generalizable to multiple effects of interest and supported by an accessible web application.

Conclusions:

• Researchers should adopt power calculation methods that explicitly account for between-study effect size variation.
• Implementing more conservative sample size determination strategies is essential for improving the replicability of psychological findings.
• Quantifying and incorporating between-study variation is recommended for robust research planning and accurate power analysis.