Data Leakage and Unverified Provenance in Public Brain Tumor MRI Datasets: A Systematic Audit

Brain tumor MRI classification studies routinely report accuracies above 99%, yet many public datasets in this domain trace to a few original sources and are often combined or cross-validated without accounting for shared origin. We audited seven brain tumor MRI datasets (34,525 images) for leakage within datasets and overlap between datasets typically treated as independent. Internal leakage was detected with perceptual hashing and rotation-aware structural similarity. Cross-dataset overlap was detected with ResNet-50 embedding similarity, checked against a random-pair baseline, a DenseNet-121 sensitivity analysis, and blinded two-rater review (all 75 sampled above-threshold pairs were true duplicates; 95% CI 95.1-100). A further blinded review just below the threshold found roughly half of the sampled pairs were also duplicates, so reported overlap is a lower bound. True-duplicate leakage between training and evaluation splits occurred in three of four datasets with predefined splits, including a peer-reviewed dataset and one documented as overlap-free. For 14 of 16 dataset pairs, 15.6-99.1% of the smaller dataset's images were automatically matched to the other, including datasets described as collected from named hospitals. In a controlled experiment on the one dataset with patient identifiers, image-level splitting yielded higher accuracy than patient-level splitting (97.10 ± 0.69% vs. 91.82 ± 2.77%; Welch's t-test, p = 0.017), with a 10.1-point drop in meningioma F1. This addresses within-dataset correlation only, so it is a conservative estimate of total inflation. Reported performance warrants cautious interpretation, and tooling for verifying dataset independence is lacking. We propose a checklist for dataset creators and researchers.