A Closer Look at Benchmarking Self-supervised Pre-training with Image Classification

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning (SSL) leverages unlabeled data for model training, reducing reliance on costly manual labeling.
• SSL is crucial in computer vision for pre-training models, enabling tasks like transfer learning and few-shot learning.
• Evaluating the quality of representations learned by SSL models across diverse downstream tasks remains a challenge.

Purpose of the Study:

• To investigate the correlation between classification-based evaluation protocols for SSL.
• To assess how well these protocols predict downstream performance across various dataset types.
• To understand the impact of model architecture and dataset domain shifts on evaluation protocol reliability.

Main Methods:

• Conducted a comprehensive study involving eleven image datasets and 26 pre-trained models using diverse SSL methods.
• Evaluated SSL methods using in-domain protocols: fine-tuning, linear probing, and k-nearest neighbors (kNN).
• Analyzed the influence of batch normalization and dataset domain shifts on protocol performance and correlation.

Main Results:

• In-domain linear probing and kNN protocols demonstrated the strongest average predictive power for out-of-domain performance.
• Found that most performance differences between discriminative and generative SSL methods are attributable to model backbone variations.
• Established that evaluation protocol robustness varies with dataset domain shifts.

Conclusions:

• Linear/kNN probing serve as effective general predictors for assessing SSL representation quality across different downstream applications.
• Model backbone architecture significantly influences performance, often more than the specific SSL training method.
• Further research is needed to refine SSL evaluation protocols for diverse real-world scenarios.