Addressing low statistical power in computational modelling studies in psychology and neuroscience

Area of Science:

• Computational modeling
• Statistical inference
• Bayesian methods

Background:

• Computational modeling aids in understanding complex data but faces challenges.
• Determining adequate sample sizes for computational studies, especially for model selection, is often overlooked.
• Bayesian model selection is a common technique for comparing alternative models.

Purpose of the Study:

• To introduce a power analysis framework for Bayesian model selection.
• To assess the statistical power of model selection in psychology and human neuroscience.
• To highlight issues with fixed effects model selection.

Main Methods:

• Developed a power analysis framework for Bayesian model selection.
• Empirically evaluated the statistical power of model selection in published studies.
• Analyzed the impact of sample size and the number of models on statistical power.

Main Results:

• Statistical power increases with sample size but decreases with more models considered.
• A significant majority of reviewed psychology and human neuroscience studies (41/52) had inadequate power (<80%) for correct model identification.
• Fixed effects model selection exhibits high false positive rates and sensitivity to outliers.

Conclusions:

• Current practices in psychology and neuroscience often lead to low statistical power in model selection.
• The reliance on fixed effects model selection poses serious statistical risks.
• The developed framework can guide researchers in designing more powerful computational studies.