ImageNet, RadiologyNET, or RadImageNet? Evaluating Pretraining Across CT, MRI, and X-Ray Classification

Due to the limited availability of annotated medical datasets, medical image classification typically relies on transfer learning from large-scale natural image datasets such as ImageNet. However, it remains unclear when domain-specific pretraining offers greater benefits. This study evaluates the impact of different pretrained weights on neural network performance in radiological image classification tasks. Four strategies were compared: ImageNet, RadiologyNET, and RadImageNet model fine-tuning against learning from randomly initialised weights (Baseline). Experiments were conducted over ten seed-paired runs using ResNet50, InceptionV3, DenseNet121, EfficientNet-B3, and MobileNetV3-Large on three classification tasks: multiclass Chest CT, multiclass BRISC brain MRI, and binary FracAtlas X-ray fracture classification. Pretraining generally improved performance over the Baseline, with the largest improvements observed on the Chest CT task. The Friedman tests identified significant differences between initialisation strategies for every architecture on all three tasks (all p≤0.0002). ImageNet achieved the best results in most architecture-task combinations, while RadiologyNET was comparable or superior in selected cases, including three of five architectures on Chest CT. RadImageNet showed less consistent transfer, particularly with DenseNet121. Pretrained models also generally reached high validation performance more rapidly than randomly initialised models. These results further support previous findings that ImageNet-pretrained models can perform comparably to or better than domain-specific radiological models when fine-tuned on sufficient downstream data, while the observed differences are consistent with possible effects of source-target alignment, downstream data availability, and network architecture.