Effect of data leakage in brain MRI classification using 2D convolutional neural networks

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Neurological Disease Diagnostics

Background:

• 2D Convolutional Neural Networks (CNNs) show promise in diagnosing neurological diseases using MRI.
• Developing generalizable CNN models is challenging due to potential data leakage during cross-validation (CV).

Purpose of the Study:

• To quantitatively assess the impact of data leakage from 2D slice-level splitting of 3D MRI data in 2D CNN models.
• To evaluate the effect of slice-level CV on classifying Alzheimer's disease (AD) and Parkinson's disease (PD).

Main Methods:

• Utilized three 2D CNN models for AD and PD classification.
• Employed 2D slice-level cross-validation on 3D MRI datasets (OASIS, ADNI, PPMI, and a local PD dataset).
• Conducted experiments with randomly labeled data to validate the extent of data leakage.

Main Results:

• Slice-level CV inflated test set accuracy by 30-55% across multiple datasets (OASIS, ADNI, PPMI, PD Versilia).
• Randomized data experiments showed ~96% accuracy with slice-level split vs. ~50% with subject-level split.
• The data leakage effect is severe, especially pronounced in smaller datasets.

Conclusions:

• Slice-level cross-validation in 2D CNNs for MRI-based neurological disease diagnosis leads to significant overestimation of performance.
• Subject-level splitting is crucial for accurate model evaluation to prevent data leakage.
• Findings highlight the critical need for appropriate data splitting strategies in neuroimaging AI research.