Multimodal sparse fusion transformer network with spatio-temporal decoupling for breast tumor classification.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Oncology

Background:

• Multimodal ultrasound imaging is crucial for breast cancer diagnosis, analyzing tumor morphology, vascularity, and stiffness.
• Manual interpretation is time-consuming and expertise-dependent, while computer-aided methods face challenges from data heterogeneity and quality variations.

Purpose of the Study:

• To develop an efficient and accurate computer-aided classification method for multimodal breast tumor analysis.
• To introduce the Multimodal Sparse Fusion Transformer Network (MSFT-Net) for robust feature fusion.

Main Methods:

• Proposed MSFT-Net utilizes a Spatio-Temporal Decoupling Attention (STDA) architecture to extract modality-specific features.
• Incorporated Mixed-Scale Convolution Module (MSCM) for multi-scale feature extraction and Sparse Cross-Attention Module (SCAM) for adaptive information fusion.
• Trained and validated on a multimodal breast tumor dataset (US, SMI, SE) from 458 patients and the BraTS'21 MRI dataset.

Main Results:

• MSFT-Net demonstrated superior performance in multimodal breast tumor classification compared to existing state-of-the-art methods.
• The network effectively disentangled and fused features from heterogeneous ultrasound modalities.
• Achieved robust classification accuracy, indicating strong generalizability.

Conclusions:

• MSFT-Net offers a novel and effective approach for multimodal breast tumor classification.
• The proposed network provides fast and reliable decision support for radiologists in breast cancer diagnosis.
• Highlights the potential of advanced AI in improving diagnostic efficiency and accuracy in medical imaging.