Reliability of foundation models for image retrieval in histopathology

Ensuring fairness and explainability is essential for the development of ethical, reliable, and effective AI systems in healthcare. Bias in AI models can contribute to disparities in clinical outcomes, challenging equity in medical decision-making. Content-Based Image Retrieval (CBIR) offers interpretable, visual tools to support diagnostic processes; however, these tools remain susceptible to biases inherent in the data. This study investigates covariate bias arising from differences in scanning devices within Foundation Models (FMs) used for CBIR in histopathology. We introduce a unique dataset comprising spatially co-registered images derived from the same histopathology slides scanned using two distinct scanners. This design enables a targeted analysis of scanner-induced variability in FM representations. Among the FMs assessed, Vision Transformer (ViT)-based architectures such as UNI, Virchow2, and GigaPath, demonstrated top performance and best generalization properties across scanners.