Does every study? Implementing ordinal constraint in meta-analysis

Area of Science:

• Psychology
• Statistics

Background:

• Meta-analysis aims to estimate true effect size across studies.
• Qualitatively different study results (e.g., varying effect directions) can render average effects uninterpretable.
• A critical first step in meta-analysis should be assessing directional consistency across studies.

Purpose of the Study:

• To propose a new statistical model for meta-analysis that incorporates ordinal constraints.
• To address the issue of uninterpretable average effect sizes arising from qualitatively different study results.
• To provide a method for determining if a single underlying mechanism can explain results across all studies.

Main Methods:

• A model with ordinal constraints (the "every study" model) is proposed.
• This model is compared against unconstrained models that allow for effects in opposite directions.
• The approach utilizes surface statistics (effect size, sample size) common in meta-analysis.

Main Results:

• If ordinal constraints hold, a single underlying mechanism may explain all study results.
• Holding ordinal constraints can lead to reduced between-study heterogeneity.
• This approach makes average effect sizes interpretable, even with initial qualitative differences.

Conclusions:

• The proposed model comparison approach, using ordinal constraints, improves meta-analysis validity.
• This method allows for interpretable average effects by ensuring directional consistency.
• The approach is illustrated with a familiar-word-recognition effect meta-analysis and includes R-code.