One size does not fit all in evaluating model selection scores for image classification.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Selecting pretrained models for image classification typically requires extensive finetuning.
• A standardized evaluation of transferability scores is lacking, hindering efficient model selection.
• Reducing computational burden in model selection is crucial for practical applications.

Purpose of the Study:

• To evaluate the effectiveness of 14 transferability scores for ranking pretrained models.
• To provide a consistent approach for balancing accuracy and efficiency in model selection.
• To guide practitioners in choosing optimal models without exhaustive finetuning.

Main Methods:

• Evaluated 14 transferability scores across 11 benchmark datasets.
• Included both Convolutional Neural Network (CNN) and Vision Transformer (ViT) models.
• Ensured consistent experimental conditions to mitigate variability from prior research.

Main Results:

• Transferability score effectiveness varied significantly based on dataset characteristics and model architectures.
• Vision Transformer (ViT) models demonstrated superior transferability, particularly on fine-grained datasets.
• No single score proved universally optimal; effectiveness was context-dependent.

Conclusions:

• The study offers insights into selecting appropriate transferability scores for optimized model selection.
• Identified scores suitable for resource-constrained environments by considering computational efficiency.
• Aims to facilitate more efficient deployment of image classification models in practice.