Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection

Area of Science:

• Cognitive Science
• Computational Neuroscience
• Psychology

Background:

• Cross-validation (CV) is widely used to evaluate and select mathematical models of cognition and behavior based on predictive performance.
• CV aims to measure model generalizability by quantifying out-of-sample predictive accuracy.
• Despite its popularity, the limitations of CV, particularly in specific applications, are often overlooked.

Purpose of the Study:

• To demonstrate the limitations of Bayesian leave-one-out cross-validation (LOO), a specific form of CV.
• To illustrate these limitations using concrete examples with idealized data.

Main Methods:

• The study employed three distinct examples to showcase the identified limitations.
• Analysis focused on the performance of LOO in scenarios with infinitely sized datasets perfectly aligned with a simple model.

Main Results:

• In all three examples, LOO provided only bounded and modest support for the simple model, despite perfect data fit.
• This indicates that LOO may not accurately reflect the true support for a model under ideal conditions.

Conclusions:

• Cross-validation (CV) is not a universally applicable panacea for model selection in cognitive science.
• The findings highlight the need for careful consideration of CV's limitations, especially Bayesian leave-one-out cross-validation (LOO), in model evaluation.