Replication is about effect size: Comment on Maxwell, Lau, and Howard (2015).

Area of Science:

• Psychology
• Statistical Methods

Background:

• Null hypothesis testing (NHT) is widely used but can provide a misleading sense of certainty.
• The reliability of single studies is often overestimated, leading to a false promise of trust.

Purpose of the Study:

• To challenge the reliance on single studies for establishing scientific truth.
• To propose a more robust approach for evaluating the replicability of research findings.
• To highlight the utility of cumulative meta-analyses in assessing effect size reproducibility.

Main Methods:

• Critiquing the traditional approach of null hypothesis testing.
• Advocating for a focus on effect size reproducibility and confidence interval precision.
• Emphasizing the application of cumulative meta-analyses for ongoing research synthesis.

Main Results:

• Single studies, particularly those relying solely on NHT, do not guarantee reliable findings.
• Replicability is better assessed by the consistency of effect sizes across studies.
• Narrowing confidence intervals indicates increasing precision and reliability of effect size estimates.

Conclusions:

• The trustworthiness of scientific findings hinges on replicability, not isolated statistical significance.
• Cumulative meta-analyses are essential tools for understanding the robustness and magnitude of psychological effects.
• Future research should prioritize methods that enhance the reproducibility of effect sizes.