Deep-learning-based HER2 status assessment from multimodal breast cancer data predicts neoadjuvant therapy response

Accurate assessment of human epidermal growth factor receptor 2 (HER2) status is crucial for effective breast cancer treatment planning and improved patient outcomes. Traditional needle biopsies, limited in tissue sampling, often lead to inaccurate assessments due to intratumoural heterogeneity. Here, to address this, we introduce the deep-learning-based HER2 multimodal alignment and prediction (MAP) model, which leverages pretreatment multimodal breast cancer images for a more comprehensive reflection of tumour characteristics and provides more accurate HER2 status prediction. We develop patient response MAP models to demonstrate the HER2 prediction performance of our model compared with needle biopsies from patients receiving neoadjuvant therapy. A large-scale multimodal breast cancer dataset from 4 centres, consisting of 14,472 images from 6,991 cases, is adopted in this study, and the results consistently demonstrate the superiority of our HER2 MAP model in predicting patient response. These findings highlight the substantial advantages of our HER2 predictions. Our study provides physicians with a crucial tool for informed clinical decisions and treatment plans, aiming to improve outcomes in patients with breast cancer.