Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach

Area of Science:

• Psychological research methodology
• Statistical power analysis

Background:

• Statistical power and sample size are crucial for research validity.
• Effect size estimates in psychology often have associated uncertainty.
• Existing sample size planning methods may lead to underpowered or overpowered studies.

Purpose of the Study:

• To introduce a novel power-calibrated effect size (PCES) approach for sample size planning.
• To address the limitations of current methods in accounting for effect size uncertainty.
• To ensure accurate statistical power through properly calibrated sample sizes.

Main Methods:

• Derivation of the PCES for various statistical comparisons (means, proportions, correlations).
• Development of a method to properly calibrate sample size planning with uncertain effect sizes.
• Application of the PCES approach to replication study design.

Main Results:

• The PCES approach provides sample sizes that are neither too small nor too large.
• This method ensures the desired level of statistical power is achieved.
• PCES is applicable to a range of common statistical tests.

Conclusions:

• The PCES approach offers a more accurate and reliable method for sample size planning in psychological research.
• This approach mitigates the risks of underpowered or overpowered studies.
• Accessible tools (website, code) are provided to facilitate the use of the PCES method.