A deep learning system for non-invasive breast cancer diagnosis with multimodal data

Early and accurate diagnosis of breast cancer is critical for minimizing needle biopsies and enhancing patient outcomes and requires effective integration of multimodal information. In this article, we introduce a breast cancer intelligent non-invasive diagnosis system (BINDS) to integrate multimodal medical imaging data for breast cancer risk assessment and subtype classification. BINDS uses a two-stage diagnostic approach to match the clinical workflow, where an initial assessment with ultrasound and/or mammography is performed, followed by a more comprehensive multimodal diagnosis incorporating magnetic resonance imaging. In addition, a new radiology-pathology alignment mechanism is proposed to facilitate extraction of pathology-relevant features from radiological images. BINDS is developed and validated with a diverse dataset of 27,048 participants from 8 centres and 7 public datasets. Importantly, BINDS supports flexible combinations of input modalities during training and validation. Notably, BINDS attains an area under the receiver operating characteristic curve of 0.973, and can assist radiologists in reducing biopsies of benign lesions by up to 32.4%. These findings highlight the potential of BINDS to advance breast cancer diagnosis by enabling precise and adaptable decision-making across diverse clinical scenarios and resource settings.