TET Loss: A Temperature-Entropy Calibrated Transfer Loss for Reliable Medical Image Classification

Deep learning models for medical image classification often exhibit overconfident predictions and domain mismatch when transferred from natural image pretraining, which undermines their generalization and clinical reliability. This study proposes TET Loss (Temperature-Entropy calibrated Transfer Loss Function), a plug-and-play objective function that combines temperature scaling to moderate logit sharpness with entropy regularization to promote uncertainty-aware learning. TET Loss is model-agnostic and introduces zero inference-time overhead. Across four public benchmarks (BreastMNIST, DermaMNIST, PneumoniaMNIST, and RetinaMNIST), TET Loss consistently enhances CNNs, transformers, and hybrid backbones under short 10-epoch fine-tuning. For example, EfficientViT-M2 improves its F1 score from 53.9 to 66.7% on BreastMNIST, and BiFormer-Tiny increases its F1 from 73.1 to 86.1% with an AUC gain to 94.1%. On PneumoniaMNIST, RMT-T3 with TET Loss reaches an F1 of 96.4% and an AUC of 99.1%, surpassing several medical-specific architectures trained for 50-150 epochs. Grad-CAM visualizations demonstrate tighter lesion localization and fewer spurious activations, reflecting improved interpretability. By calibrating confidence while preserving discriminative learning, TET Loss provides a lightweight and effective pathway toward more reliable and robust medical imaging systems. Our code will be available at https://github.com/JEFfersusu/TET_loss .