Benchmarking Vision Encoders for Image Classification in Ophthalmology

Foundation vision encoders are rapidly emerging as the standard for retinal artificial intelligence. Yet, ophthalmology still lacks a comprehensive benchmark, leaving model selection for basic science and clinical translation as guesswork. Here, we present a large-scale comparison of 34 pretrained encoders on 39 classification tasks covering color fundus photography, optical coherence tomography, scanning laser ophthalmoscopy, and ultrawidefield imaging. Using a unified pipeline, we compare frozen-feature evaluation, linear probing, and end-to-end fine-tuning to determine which models translate into strong downstream performance. We show that ophthalmic transfer is highly task dependent: no single encoder dominates, and model rankings vary across datasets. Contrary to common expectations, retina-specific pretraining does not confer an advantage. Instead, several natural-image and cross-domain medical encoders match or surpass ophthalmology-specialized models, with the histopathology-pretrained Virchow achieving the strongest overall performance. In addition, pathology-pretrained encoders consistently place near the top, revealing the value of cross-domain pretraining for ophthalmic applications. We further show that inexpensive proxy evaluations are unreliable substitutes for full fine-tuning. Across fairness analyses, all encoders exhibit similar age- and sex-associated performance gaps, and larger models appear more sensitive to suboptimal learning rates, whereas smaller encoders are robust. Together, these findings provide an objective reference for encoder selection in ophthalmology and show that reliable retinal artificial intelligence depends not only on model scale or domain-specific pretraining but also on careful, protocol-aware evaluation. By releasing our code, splits, and benchmarking pipeline, we aim to establish a transparent foundation for future ophthalmic foundation-model research.