ViT-FuseNet: Same-Patient MRI-Pathology Feature Fusion for Multimodal Breast Cancer Diagnosis

Area of Science:

• Medical Imaging
• Artificial Intelligence in Oncology
• Computational Pathology

Background:

• Radiological imaging offers structural insights but lacks definitive diagnostic power for breast cancer.
• Pathological examinations are crucial for conclusive breast cancer diagnosis.
• Integrating diverse imaging data can enhance diagnostic capabilities.

Purpose of the Study:

• To develop and evaluate a Vision Transformer (ViT)-based multimodal fusion approach for breast cancer diagnosis.
• To combine magnetic resonance imaging (MRI) and pathology images for improved diagnostic accuracy.
• To assess the efficacy of different ViT architectures and classifiers in multimodal fusion.

Main Methods:

• A Vision Transformer (ViT)-based model was developed for fusing MRI and pathology images.
• Models utilized ViT-16 and ViT-32 architectures with various classifiers.
• Performance was rigorously evaluated using metrics like accuracy, F1 score, sensitivity, precision, ROC, AUC, and Precision-Recall (PR) curves.

Main Results:

• Multimodal image fusion models significantly outperformed single-modality models in accuracy, precision, and sensitivity.
• The ViT-B/32 (Fusion) + SVM model achieved the highest accuracy (92.53%) and PR-AP (0.9708).
• Fusion models demonstrated superior effectiveness in breast cancer diagnosis.

Conclusions:

• Evaluating both radiological and pathological images enhances diagnostic accuracy and reliability.
• Multimodal image fusion is an effective strategy for differentiating malignant from benign breast lesions.
• ViT-based fusion represents a promising advancement in breast cancer diagnostics.