Decision qualities of Bayes factor and p value-based hypothesis testing

Area of Science:

• Statistics
• Psychological Research Methods

Background:

• Null hypothesis significance testing (NHST) using p-values is a standard statistical approach.
• The Bayes factor (BF) offers an alternative method for hypothesis testing and model comparison.
• Understanding the comparative performance of BF and NHST is crucial for robust scientific inference.

Purpose of the Study:

• To compare the decision qualities of the Bayes factor (BF) method against p-value based null hypothesis significance testing (NHST).
• To assess the performance of BF and NHST using false-positive rates, true-positive rates, false-discovery rates, and posterior probabilities of the null hypothesis.
• To evaluate these methods across independent-samples t-test and ANOVA models with random factors.

Main Methods:

• A simulation study was conducted to compare BF and NHST.
• Performance metrics included false-positive rates, true-positive rates, and false-discovery rates.
• Two statistical models were used: independent-samples t-test and a two-random-factor ANOVA model.

Main Results:

• The common BF > 3 criterion is more conservative than NHST's alpha = .05, aligning better with alpha = .01.
• Increasing sample size impacts false-positive and false-discovery rates differently for BF and NHST.
• BF's false-positive and true-positive rates are sensitive to the scale factor with small effect sizes.
• Posterior probabilities of the null hypothesis can be unexpectedly high with NHST, unlike ideal BF outcomes.

Conclusions:

• The Bayes factor method provides a more conservative approach to statistical inference compared to traditional NHST.
• BF's sensitivity to scale factors necessitates careful consideration, especially in scenarios with small effect sizes.
• Findings were consistent across both t-test and ANOVA models, highlighting the generalizability of the observed differences.