Cross-Center Vision-Language Transformer for Robust Mammography-Based Breast Cancer Diagnosis

While promising results have been demonstrated for deep learning-based breast cancer diagnosis using mammography, problems persist in approaches that rely primarily on visual information. These problems include inadequate performance across diverse clinical centers, various imaging protocols, scanner types, and patient distributions. Here, we introduce Cross-Center Vision-Language Transformer (CC-VLT), a framework that integrates mammograms and clinical text to enable more robust, guided diagnosis. The framework incorporates a vision transformer for mammograms, a text transformer for salient clinical descriptors, bi-directional cross-modal attention for semantics, and a cross-center feature regularization approach to address the challenge of inter-institutional domain shifts. The framework is tested on a leave-one-center-out basis across several public mammography datasets and significantly outperforms strong baseline models in both intra- and cross-center evaluations. Our framework achieved an accuracy of 90.7% with an intra-center ROC-AUC of 0.951 and cross-center ROC-AUC results of 0.912, 0.927, and 0.934 on the CBIS-DDSM, INbreast, and VinDr-Mammo datasets, respectively. Reliability of the malignancy probability predictions improved, as evidenced by a diminished Expected Calibration Error and Brier Score. Our framework, by designing an effective integrated vision-language interaction model and implementing a cross-center feature regularization approach, sets a benchmark for robust breast cancer diagnosis across diverse clinical environments.