Rethinking Architectural Complexity in Deep Vision Models for Histopathological Image Classification

Recent advances in deep learning for histopathological image analysis have led to increasingly complex architectures that require substantial computational resources and large datasets. While these models achieve strong performance, they often fail to deliver meaningful benefits to clinical practitioners. This study provides a systematic empirical analysis of performance-efficiency trade-offs among CNN and transformer architectures under varying data regimes in histopathological image classification. To investigate this, we conducted a comprehensive evaluation of widely adopted convolutional and attention-based models across three distinct histopathological tissue classification datasets. Model performance was further assessed using standard and clinically relevant diagnostic metrics. Across datasets, EfficientNet-B0 and ResNet-50 achieved near-peak performance with limited data (F1 $\approx$ 0.983 at 10%) and 2-5 $\times$ faster training ( $\approx$ 638-675 s vs. 1903-3168 s), while transformers offered modest gains mainly on imbalanced data (F1 $\approx$ 0.865-0.964) at substantially higher compute. Statistical analyses confirmed the significance of the observed trends. Based on these findings, we derive a set of empirical recommendations to assist architecture selection according to data availability, class balance, and computational constraints, with an emphasis on practical feasibility in resource-constrained settings. We emphasise that these recommendations are derived from controlled benchmark experiments and require independent validation in real-world clinical settings.