A Multimodal Multitask Deep Learning Model Based on Ultrasound RF Signals for Joint Assessment of Breast Masses and...

Area of Science:

• Medical Imaging
• Artificial Intelligence in Oncology
• Breast Cancer Diagnostics

Background:

• Evaluating lymph node status and distinguishing benign from malignant breast masses are critical clinical challenges.
• Deep learning models offer potential for improving diagnostic accuracy in breast cancer assessment.

Purpose of the Study:

• To develop a multimodal deep learning model using radio frequency (RF) data for simultaneous breast mass classification and lymph node status assessment.
• To integrate Class Activation Mapping (CAM) heatmaps to assist ultrasonographers and explore the correlation between heatmaps and neutrophil distribution.

Main Methods:

• A deep learning model utilizing ResNet-18 and a multi-task loss (MTL) function was trained on B-mode and RF data from 308 breast cancer patients.
• Class Activation Mapping (CAM) heatmaps were generated and analyzed, alongside animal experiments to examine neutrophil distribution in relation to heatmap regions.

Main Results:

• The optimal model combining B-mode and RF data with MTL achieved high diagnostic performance (AUCs ranging from 0.922 to 0.967).
• CAM heatmaps showed distinct patterns for metastatic versus non-metastatic lymph nodes, and assisted sonographers improved diagnostic AUCs.
• Animal studies confirmed a correlation between heatmap regions and neutrophil distribution, with higher concentrations in metastatic cases.

Conclusions:

• A multimodal deep learning model using RF data and CAM heatmaps effectively aids in breast cancer assessment, particularly for lymph node status.
• The findings suggest a link between neutrophil infiltration and the model's focus areas, offering insights into underlying physiological mechanisms.