Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian analysis of variance

Area of Science:

• Cognitive Science
• Statistics
• Psychology

Background:

• Bayesian linear mixed-effects models (LMMs) and Bayesian analysis of variance (ANOVA) are common in cognitive sciences for null hypothesis testing.
• Bayes factor software is accessible, but correct data and model specification remain unclear.
• Many researchers aggregate data by subject for Bayesian analyses.

Purpose of the Study:

• To demonstrate the problems of null hypothesis tests on aggregated data in Bayesian analysis.
• To evaluate the impact of violated assumptions on Bayes factor results.
• To provide recommendations for accurate Bayesian inference.

Main Methods:

• Simulation-based calibration for model inference.
• Application to several example experimental designs.
• Comparison of analyses on aggregated versus nonaggregated data.

Main Results:

• Null hypothesis tests on aggregated data can be problematic in Bayesian analysis, mirroring issues in frequentist approaches.
• When sphericity is violated, Bayes factors on aggregated data are too conservative or too liberal depending on contrast variance.
• Ignoring random item slope variance in aggregated data analyses leads to biased (too liberal) Bayes factors.

Conclusions:

• Aggregating data by subject for Bayesian null hypothesis tests can lead to biased Bayes factors.
• Bayesian LMMs on nonaggregated (individual trial) data, with explicit modeling of random effects, circumvent these issues.
• Accurate Bayesian inference requires careful data and model specification, avoiding data aggregation when possible.