Workflow techniques for the robust use of bayes factors

Area of Science:

• Cognitive Science
• Bayesian Statistics
• Computational Statistics

Background:

• Bayes factors are widely used in cognitive sciences for hypothesis comparison.
• Their accuracy is sensitive to data/model assumptions and computational implementation details.

Purpose of the Study:

• To investigate the behavior and potential misbehavior of Bayes factors under various conditions.
• To assess the accuracy and bias of Bayes factor estimation using bridge sampling.
• To evaluate the stability and decision variability associated with Bayes factors.

Main Methods:

• Simulation-based calibration was employed to test Bayes factor estimation accuracy and bias.
• Stability was assessed against Markov Chain Monte Carlo (MCMC) draws and data sampling variations.
• A utility function was used to examine decision variability.

Main Results:

• Identified specific conditions under which Bayes factors can be inaccurate or biased.
• Demonstrated the impact of computational implementation (bridge sampling) on estimation.
• Quantified the variability in hypothesis testing decisions based on Bayes factors.

Conclusions:

• Bayes factors require careful implementation and validation for reliable hypothesis testing.
• Researchers should utilize a structured workflow to assess the robustness of Bayes factors in their analyses.
• Understanding Bayes factor limitations is crucial for advancing cognitive science research.