Do We Train on Test Data? Purging CIFAR of Near-Duplicates

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• CIFAR-10 and CIFAR-100 are standard benchmarks for evaluating deep learning models in computer vision.
• Existing CIFAR test sets contain duplicate images present in their training sets, potentially biasing performance evaluations.

Purpose of the Study:

• To address the bias caused by duplicate images in CIFAR test sets.
• To create a "fair CIFAR" (ciFAIR) dataset by replacing duplicate test images.
• To re-evaluate deep learning model performance on a duplicate-free dataset to assess generalization capabilities.

Main Methods:

• Identified and replaced duplicate images in CIFAR-10 and CIFAR-100 test sets with new, in-domain samples.
• Kept the original training sets unchanged to preserve pre-trained model validity.
• Re-evaluated state-of-the-art Convolutional Neural Network (CNN) architectures on the ciFAIR dataset.

Main Results:

• A significant decrease in classification accuracy (9-14%) was observed on the ciFAIR dataset compared to the original CIFAR test sets.
• The findings suggest that current deep learning models may be overfitting to memorizing training data rather than learning abstract visual concepts.
• The ciFAIR dataset and associated pre-trained models are publicly released.

Conclusions:

• The presence of duplicate images in standard CIFAR benchmarks inflates reported model performance.
• A substantial gap exists between memorization and true generalization capabilities in current deep learning models.
• The ciFAIR dataset provides a more reliable benchmark for assessing the generalization ability of image recognition models.