Meta-analysis to integrate effect sizes within an article: Possible misuse and Type I error inflation

Area of Science:

• Psychology
• Research Methodology
• Statistics

Background:

• Internal meta-analysis, combining a researcher's own studies, is increasingly used.
• This method offers narrower confidence intervals but poses potential risks.
• Motivated researchers may exploit this for significant outcomes.

Purpose of the Study:

• To investigate the impact of internal meta-analysis on false-positive rates.
• To demonstrate how motivated use can inflate statistical significance.
• To provide recommendations for responsible application of internal meta-analysis.

Main Methods:

• Conducted Monte Carlo simulations to model the effects.
• Performed a literature review to assess awareness and prevalence.
• Analyzed decision-making processes based on statistical significance.

Main Results:

• Internal meta-analysis can significantly inflate false-positive rates when decisions to continue studies are based on significance.
• The probability of false positives increases beyond nominal levels under these conditions.
• Literature review indicated varying awareness of this methodological pitfall.

Conclusions:

• Researchers must exercise caution when using internal meta-analysis, especially when making sequential decisions.
• Implementing safeguards is crucial to prevent inflated false-positive rates.
• Recommendations are provided to ensure the integrity of internal meta-analysis findings.