Enhancing Breast Lesion Detection in Mammograms via Transfer Learning

Early detection of breast cancer via mammography enhances patient survival rates, prompting this study to assess object detection models-Cascade R-CNN, YOLOv12 (S, L, and X variants), RTMDet-X, and RT-DETR-X-for detecting masses and calcifications across four public datasets (INbreast, CBIS-DDSM, VinDr-Mammo, and EMBED). The evaluation employs a standardized preprocessing approach (CLAHE, cropping) and augmentation (rotations, scaling), with transfer learning tested by training on combined datasets (e.g., INbreast + CBIS-DDSM) and validating on held-out sets (e.g., VinDr-Mammo). Performance is measured using precision, recall, mean Average Precision at IoU 0.5 (mAP50), and F1-score. YOLOv12-L excels in mass detection with an mAP50 of 0.963 and F1-score up to 0.917 on INbreast, while RTMDet-X achieves an mAP50 of 0.697 on combined datasets with transfer learning. Preprocessing improves mAP50 by up to 0.209, and transfer learning elevates INbreast performance to an mAP50 of 0.995, though it incurs 5-11% drops on CBIS-DDSM (0.566 to 0.447) and VinDr-Mammo (0.59 to 0.5) due to domain shifts. EMBED yields a low mAP50 of 0.306 due to label inconsistencies, and calcification detection remains weak (mAP50 < 0.116), highlighting the value of high-capacity models, preprocessing, and augmentation for mass detection while identifying calcification detection and domain adaptation as key areas for future investigation.