A Meta-Meta-Analysis: Empirical Review of Statistical Power, Type I Error Rates, Effect Sizes, and Model Selection of...

Area of Science:

• Psychological research methodology
• Quantitative psychology
• Statistical analysis

Background:

• Meta-analyses are crucial for synthesizing psychological research.
• Previous reviews highlighted potential issues in meta-analytic practices.
• Understanding statistical power and error rates is vital for reliable conclusions.

Purpose of the Study:

• To examine statistical power, Type I error rates, and model selection in psychological meta-analyses.
• To assess the quality of meta-analyses published between 1995 and 2005.
• To provide recommendations for enhancing future meta-analytic research.

Main Methods:

• Analysis of meta-analyses published in Psychological Bulletin (1995-2005).
• Retrospective and prospective power estimations.
• Examination of Type I error probabilities from multiple comparisons.
• Assessment of heterogeneity using Birge's ratio.
• Comparison of fixed-effects and random-effects model usage.

Main Results:

• Low statistical power was observed for univariate moderators and tests of residual variability.
• A sizable estimated probability of Type I error was found due to numerous significance tests.
• Median effect sizes were smaller than conventional medium effect sizes.
• Typical meta-analyses exhibited heterogeneity exceeding sampling error alone.
• Fixed-effects models were more common than random-effects models, though the latter increased over time.

Conclusions:

• Many psychological meta-analyses exhibit methodological limitations impacting result reliability.
• Researchers must carefully consider statistical power, error rates, and heterogeneity.
• Adoption of appropriate statistical models and prospective power calculations is recommended.
• Findings inform best practices for conducting and interpreting future meta-analyses.