Comparative Study of CNN, Hybrid, and Transformer Architectures in Medical Image Classification

Deep learning has significantly advanced medical image analysis, yet the relative strengths of convolutional neural networks (CNNs), vision transformers (ViTs), and hybrid CNN-Transformer models remain unclear. We present a systematic comparison of these model families across six 2D MedMNIST classification tasks under a controlled parameter budget of 20 to 30 million parameters, using both full fine-tuning and linear-probing. Among all evaluated models, the self-supervised CNN DINO-S achieves the highest average accuracy and AUC, while hybrid models such as MedViT-S perform comparably by combining local feature extraction with global context modeling. In contrast, compact ViTs generally underperform CNNs and hybrids under full fine-tuning when model size is controlled, and their performance depends strongly on pretraining strategy. Supervised medical pretraining on NIH ChestX-ray14 and CheXpert yields modest, task-dependent gains but does not alter the overall ranking of model families. Large-scale foundation models, including MedMAE, Rad-DINO, and CheXFound, also underperform smaller fully fine-tuned models under linear-probing, underscoring the importance of domain and task alignment. Overall, our results show that larger models and larger pretraining datasets do not automatically lead to better downstream performance. Instead, transfer effectiveness in medical imaging is driven primarily by architectural inductive biases, pretraining strategy, and domain relevance.