Transformer-based Deep Learning Models with Shape Guidance for Predicting Breast Cancer in Mammography Images

Recent breast cancer research has investigated shape-based attention guidance in Vision Transformer (ViT) models, focusing on anatomical structures and the heterogeneity surrounding tumors. However, few studies have clarified the optimal transformer encoder layer stage for applying attention guidance. Our study aimed to evaluate the effectiveness of shape-guidance strategies by varying the combinations of encoder layers that guide attention to breast structures and by comparing the proposed models with conventional models. For the shape-guidance strategy, we applied breast masks to the attention mechanism to emphasize spatial dependencies and enhance the learning of positional relationships within breast anatomy. We then compared the representative models-Masked Transformer models that demonstrated the best performance across layer combinations-with the conventional ResNet50, ViT, and SwinT V2. In our study, a total of 2,436 publicly available mammography images from the Chinese Mammography Database via The Cancer Imaging Archive were analyzed. Three-fold cross-validation was employed, with a patient-wise split of 70% for training and 30% for validation. Model performance on differentiating breast cancer from non-cancer images was assessed by the area under the receiver-operating characteristic curve (AUROC). The results showed that applying masks at the Shallow and Deep stages gave the highest AUROC for Masked ViT. The Masked ViT achieved an AUROC of 0.885 [95% confidence interval: 0.849-0.918], a sensitivity of 0.876, and a specificity of 0.802, outperforming all other conventional models. These results indicate that incorporating mask guidance into particular Transformer encoders promotes representation learning, highlighting their potential as decision-support tools in breast cancer diagnosis.