Data leakage in deep learning studies of translational EEG

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Deep neural networks (DNNs) are increasingly used with electroencephalography (EEG) for disease identification.
• Common practice involves segment-based data splitting, allowing data from the same subject in both training and testing sets.

Purpose of the Study:

• To investigate if segment-based holdout in DNN-EEG studies leads to overestimated classification performance.
• To compare segment-based holdout with subject-based holdout for evaluating DNN generalization.

Main Methods:

• Implemented DNN classifiers using both segment-based and subject-based holdout strategies.
• Evaluated classifier performance on two datasets: Alzheimer's disease and epileptic seizures.
• Surveyed existing literature on DNN-EEG studies to determine common data splitting practices.

Main Results:

• Subject-based holdout revealed significantly lower performance compared to segment-based holdout.
• Performance on previously unseen subjects was strongly overestimated using segment-based holdout.
• The majority of translational DNN-EEG studies employ segment-based holdout, suggesting widespread overestimation.

Conclusions:

• Segment-based data splitting in DNN-EEG research can lead to a dramatic overestimation of true classification performance.
• Subject-based holdout is a more reliable method for assessing the generalizability of DNN models in clinical EEG applications.
• Future DNN-EEG studies should adopt subject-based holdout to ensure accurate reporting of diagnostic capabilities.