Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines

Area of Science:

• Neuroimaging
• Machine Learning
• Brain-Computer Interfaces

Background:

• Decoding brain data requires robust evaluation of predictive accuracy.
• Cross-validation is essential for both performance assessment and hyper-parameter tuning of decoding models.

Purpose of the Study:

• To review and critically evaluate cross-validation procedures for neuroimaging decoding.
• To provide theoretical insights and practical recommendations for reliable decoding analysis.

Main Methods:

• Comprehensive review of cross-validation techniques.
• Empirical evaluation using diverse neuroimaging datasets (fMRI, MEG) and simulations.
• Comparison of 'leave-one-out' versus repeated random splits.

Main Results:

• The 'leave-one-out' cross-validation strategy yields unstable and biased decoding estimates.
• Repeated random splits offer more reliable and accurate performance evaluation.
• Cross-validation in neuroimaging often exhibits large confidence intervals (around 10%).
• Nested cross-validation can prevent bias during hyper-parameter tuning, but default parameters may suffice for non-sparse decoders.

Conclusions:

• Choosing appropriate cross-validation methods is crucial for accurate neuroimaging decoding.
• Repeated random splits are preferred over 'leave-one-out' for robust decoding evaluation.
• Careful consideration of cross-validation strategies, including nested approaches or default parameters, is necessary for reliable brain data analysis.