A Bayesian nonparametric test of significance chasing biases

Area of Science:

• Biostatistics
• Meta-analysis
• Research Integrity

Background:

• Concerns exist regarding statistical significance pursuit distorting research literature.
• Ioannidis and Trikalinos (2007) proposed an omnibus test for significance chasing (SC) bias.
• Existing methods may not adequately account for study diversity and heterogeneous effect sizes.

Purpose of the Study:

• To develop a Bayesian nonparametric (BNP) meta-regression model for detecting SC bias.
• To create a novel BNP test capable of diagnosing SC bias at the individual study level.
• To enhance the assessment of research integrity by accounting for study-level covariates and heterogeneity.

Main Methods:

• Developed a Bayesian nonparametric (BNP) meta-regression model.
• Introduced a new BNP test for SC bias based on predictive distribution of study power.
• The test compares significant outcome indicators against posterior predictive distributions, conditional on study covariates.

Main Results:

• The BNP model and test were illustrated using three meta-analytic datasets.
• A simulation study demonstrated the model's performance.
• The approach successfully diagnoses SC bias at the individual study level, accounting for covariates and heterogeneity.

Conclusions:

• The developed BNP model and test offer a flexible and powerful tool for detecting SC bias.
• This method improves upon existing approaches by considering study diversity and individual study characteristics.
• The findings contribute to enhancing the reliability and transparency of published research literature.