The use of meta-analytic statistical significance testing

Area of Science:

• Statistics
• Meta-analysis
• Research methodology

Background:

• Meta-analysis multiplicity, the performance of multiple statistical significance tests within a single review, is an under-researched area.
• Existing meta-analyses frequently employ statistical significance testing without addressing the implications of multiple comparisons.
• This oversight can lead to an increased risk of Type I errors and affect the reliability of meta-analytic findings.

Purpose of the Study:

• To investigate the impact of Type I errors in meta-analysis multiplicity.
• To explore how multiplicity corrections influence statistical power in meta-analyses.
• To propose guidelines for meta-analysts on handling multiple statistical significance tests.

Main Methods:

• Conducted a meta-review of 130 meta-analyses from leading education and psychology journals.
• Analyzed the extent to which statistical significance testing was used without considering multiplicity.
• Examined the application, or lack thereof, of multiplicity corrections in the reviewed studies.

Main Results:

• A significant reliance on statistical significance testing was observed across the reviewed meta-analyses.
• The majority of studies failed to acknowledge or address Type I error inflation due to multiple testing.
• There was a notable absence of multiplicity correction methods in the analyzed literature.

Conclusions:

• Meta-analysts must proactively consider issues related to multiplicity before conducting reviews.
• Failure to address Type I errors and apply multiplicity corrections can compromise the validity of meta-analytic conclusions.
• Implementing appropriate statistical practices is crucial for ensuring robust and reliable research synthesis.