Comparing researchers' degree of dichotomous thinking using frequentist versus Bayesian null hypothesis testing

Area of Science:

• Social and Behavioural Sciences
• Statistical Inference
• Psychological Research Methods

Background:

• Hypothesis testing is fundamental in social and behavioural sciences.
• Interpreting statistical evidence like p-values and Bayes factors is crucial for researchers.
• Understanding how statistical evidence influences belief in effects is important.

Purpose of the Study:

• To investigate the relationship between statistical evidence and researchers' belief in a positive effect.
• To examine the 'cliff effect'—a sharp drop in belief around statistical thresholds (e.g., p=0.05).
• To compare this effect for p-values versus Bayes factors and assess the impact of presentation mode.

Main Methods:

• A study involving 139 participants (N=139) was conducted.
• Participants' degree of belief in a positive effect was measured against statistical evidence.
• The study compared p-values and Bayes factors, and two presentation modes (implicit vs. explicit functional form).

Main Results:

• A higher proportion of 'cliff effects' were observed in conditions using p-values compared to Bayes factors.
• No clear evidence was found that the presentation mode influenced the proportion of cliff effects.
• The study provides insights into how researchers interpret different statistical metrics.

Conclusions:

• The interpretation of statistical evidence, particularly p-values, may lead to a 'cliff effect' in researchers' beliefs.
• Bayes factors appear less prone to this sharp, threshold-based shift in belief compared to p-values.
• Further research may be needed to clarify the impact of presentation modes on statistical interpretation.