A Multimodal Multitask Deep Learning Model Based on Ultrasound RF Signals for Joint Assessment of Breast Masses and...

Developing a deep learning model to simultaneously evaluate lymph node status and distinguish between benign and malignant breast masses has been a challenging clinical task. This study aimed to use radio frequency (RF) signal data to create a deep learning, multimodal multitasking model, incorporating Class Activation Mapping (CAM) heatmaps to assist ultrasonographers in assessing the overall patient state based on their expertise. Research has shown a correlation between lymph node metastases and the heterogeneity of neutrophil distribution in malignancies. Therefore, the study also aimed to analyze the correlation between CAM heatmaps and neutrophil distribution, exploring the physiological mechanisms underlying the model.

A total of 308 eligible breast cancer patients with B-mode data, RF data and ultrasound reports were selected as the training set (n = 246) and validation set (n = 62) at the Second Affiliated Hospital of Harbin Medical University from September 2018 to October 2019. Using ResNet-18 as a feature extraction network, a multi-task loss (MTL) function to design a model for breast mass classification and axillary lymph node status assessment. The model's CAM heatmaps were analyzed and integrated with the sonographer's expertise to evaluate the patient's condition. Sensitivity, specificity, accuracy and receiver operating characteristic (ROC) curve analyses were calculated. In the animal experiment, CAM heatmaps from mouse transplanted tumors were analyzed, and differences in neutrophil distribution at different heatmap color locations were examined.

The B + RF mode data with MTL was the optimal combination for modeling. The area under the ROC curve (AUC) for the DLMMRF analysis of breast mass prediction was 0.967, for lymph node status prediction was 0.922 and for patient overall status prediction was 0.944. In CAM heatmaps, the red portion for patients with lymph node metastases focused on mass margins, while for those without metastases, it centered on the mass (p < 0.001). The AUC for sonographers with CAM assistance in breast mass prediction was 0.901, for lymph node status prediction was 0.874 and for patient overall status prediction was 0.887. Animal experiments showed that heatmap patterns correlated with neutrophil distribution in tumors, with a larger neutrophil-positive area in the red section than in the blue section in both metastatic and non-metastatic groups, and a greater neutrophil-positive area in the metastatic group's red section compared to the non-metastatic group.

This study developed a multimodal multitasking deep learning model using RF data to generate CAM heatmaps, assisting ultrasonographers in assessing patient status, especially lymph nodes. The CAM heatmaps' red regions displayed higher neutrophil concentrations than the blue areas, with the metastatic group showing more neutrophils in the red regions compared to the non-metastatic group, suggesting a possible association between neutrophil infiltration and the model's attention regions.