Diagnostic Performance and Calibration of AI Models for Thyroid Ultrasound Malignancy Classification: From TN5000 to ThyroidXL

We evaluated cross-dataset transportability of three frozen encoders for binary benign-versus-malignant thyroid ultrasound classification using expert-defined nodule regions and one external cohort. Logistic regression classifiers using DINOv2, TinyUSFM, and ResNet18 features were developed on 3500 TN5000 images, calibrated on 477 duplicate-clean validation images and tested on 958 images. External evaluation used maximum image scores from 2083 ThyroidXL images in 739 patients with cytological or histopathological outcomes. No external retraining, recalibration or threshold adjustment was performed. Additional analyses evaluated four classifiers, encoder-recommended preprocessing and calibration intercepts and slopes. Internal areas under the receiver operating characteristic curve (AUROCs) were 0.9081 for TinyUSFM, 0.8946 for DINOv2, and 0.8896 for ResNet18. Primary external AUROCs were 0.8464 (95% confidence interval, 0.8164-0.8731), 0.7954, and 0.7833, respectively. Sensitivities at frozen high-sensitivity thresholds were 0.9972, 0.9717, and 0.9887, but specificities were 0.1736, 0.1891, and 0.1373. External calibration intercepts were negative; slopes were 0.891, 0.645, and 0.697. DINOv2 ranked first externally with four of five classifiers. With recommended preprocessing, ResNet18 ranked first numerically (0.8329 versus 0.8214 for DINOv2); their paired difference was inconclusive. Encoder rankings depended on the classifier and preprocessing. These results apply to the three evaluated implementations, not pretraining strategies generally. Transported patient scores overestimated malignancy and had low specificity. Prospective assessment is needed before clinical use.