Quantifying convergence and consistency

Area of Science:

• Scientific methodology
• Reproducibility in science
• Meta-analysis

Background:

• The reproducibility crisis reveals limitations in current scientific evaluation methods.
• Existing meta-analysis techniques assess evidence consistency but not convergence across different empirical approaches.
• A gap exists in quantifying how diverse study types collectively support a hypothesis.

Purpose of the Study:

• To introduce the cumulative evidence index (CEI), a novel metric for meta-analysis.
• To quantify both the consistency and convergence of evidence for causal hypotheses.
• To provide a more holistic assessment of scientific evidence by integrating multiple study types.

Main Methods:

• Developed the cumulative evidence index (CEI) using Bayesian statistics.
• The CEI model quantifies evidence from four study types: positive/negative intervention and positive/negative non-intervention.
• Assesses credence for three causal relations: excitatory, inhibitory, or no-connection.

Main Results:

• The CEI provides a quantitative measure of evidence convergence alongside consistency.
• It integrates findings from diverse empirical methods to evaluate causal hypotheses.
• The CEI offers a more comprehensive perspective on evidence than traditional measures like p-values.

Conclusions:

• The CEI addresses the reproducibility crisis by formally assessing convergent evidence.
• It demonstrates how diverse study types can build scientific consensus, even with inconsistent individual results.
• The CEI quantitatively captures scientists' qualitative assessment of epistemic principles.