Beyond overall effects: A Bayesian approach to finding constraints in meta-analysis.

Area of Science:

• Psychological Research Methods
• Statistical Analysis
• Behavioral Science

Background:

• Traditional meta-analyses often focus on average effects, which can be misleading due to variations in study designs and methodologies.
• The construct of a meta-analytic mean may not always be defensible when study distributions are influenced by researcher choices.
• An alternative approach is needed to evaluate the consistency of findings across individual studies within a meta-analysis.

Purpose of the Study:

• To propose an alternative goal for meta-analysis: determining if observed relations are stable across all included studies.
• To introduce four statistical models for assessing the directionality and consistency of true effects across studies.
• To demonstrate the utility of Bayes factor model comparison for evaluating these models in empirical meta-analyses.

Main Methods:

• Development of four distinct statistical models: (a) all studies null, (b) single true nonzero effect, (c) all true effects in the same direction, and (d) mixed positive and negative true effects.
• Application of Bayes factor model comparison to differentiate between the proposed models.
• Empirical testing of the proposed methodology on four existing meta-analyses.

Main Results:

• The proposed models and Bayes factor approach provide a framework for assessing the stability of relations across studies.
• Demonstrated the practical application and usefulness of this alternative meta-analytic goal in real-world research scenarios.
• The analysis of extant meta-analyses highlighted the importance of considering effect stability beyond just average effects.

Conclusions:

• Meta-analysis can be advanced by focusing on the stability of relations across studies, rather than solely on average effects.
• The proposed models and Bayes factor comparison offer a robust method for evaluating effect consistency.
• This approach enhances the interpretability and defensibility of meta-analytic findings by examining directional stability.