Related Experiment Video
Updated: Jul 23, 2025
06:34
A Single-Channel and Non-Invasive Wearable Brain-Computer Interface for Industry and Healthcare
Published on: July 7, 2023
2.4K
k-Fold Cross-Validation Can Significantly Over-Estimate True Classification Accuracy in Common EEG-Based Passive BCI
Jacob White1, Sarah D Power1,2
1Faculty of Engineering and Applied Science, Memorial University of Newfoundland, St. John's, NL A1B 3X5, Canada.
Sensors (Basel, Switzerland)
|July 14, 2023
Summary
K-fold cross-validation (CV) in passive Brain-Computer Interface (BCI) studies can inflate accuracy estimates due to sample autocorrelation. Researchers should minimize samples per trial and report both k-fold and block-wise CV results for reliable mental state classification.
Area of Science:
- Neuroscience
- Machine Learning
- Biomedical Engineering
Background:
- Passive Brain-Computer Interface (BCI) studies often segment long trials into epochs for classification.
- K-fold cross-validation (CV) is commonly used despite known issues with sample autocorrelation from the same trial.
- The precise impact of autocorrelation on k-fold CV estimates in passive BCI remains unclear.
Purpose of the Study:
- To investigate how sample correlation within classes affects EEG-based mental state classification accuracy estimated by k-fold CV.
- To compare k-fold CV results against ground-truth (GT) accuracy and block-wise CV.
- To explore the influence of class separability, feature sets, and classifiers on these estimations.
Main Methods:
- Designed a novel experiment to manipulate the degree of correlation among samples within a class.
- Employed k-fold CV and block-wise CV for accuracy estimation.
- Compared estimated accuracies against a ground-truth (GT) accuracy measure.
- Investigated varying degrees of true class separability, feature sets, and classifiers.
Main Results:
- K-fold CV demonstrated inflated classification accuracy, overestimating ground-truth by up to 25% under certain conditions.
- Block-wise CV, while intended to mitigate autocorrelation, underestimated ground-truth accuracy by up to 11%.
- The degree of autocorrelation significantly impacted the reliability of k-fold CV estimates.
Conclusions:
- K-fold CV can provide misleadingly high accuracy in passive BCI due to autocorrelation.
- Block-wise CV also presents inaccuracies, potentially underestimating true performance.
- Recommendations include reducing samples per trial and reporting both k-fold and block-wise CV for robust interpretation of passive BCI results.
