A multimodal machine learning model for the stratification of breast cancer risk

Machine learning models for the diagnosis of breast cancer can facilitate the prediction of cancer risk and subsequent patient management among other clinical tasks. For the models to impact clinical practice, they ought to follow standard workflows, help interpret mammography and ultrasound data, evaluate clinical contextual information, handle incomplete data and be validated in prospective settings. Here we report the development and testing of a multimodal model leveraging mammography and ultrasound modules for the stratification of breast cancer risk based on clinical metadata, mammography and trimodal ultrasound (19,360 images of 5,216 breasts) from 5,025 patients with surgically confirmed pathology across medical centres and scanner manufacturers. Compared with the performance of experienced radiologists, the model performed similarly at classifying tumours as benign or malignant and was superior at pathology-level differential diagnosis. With a prospectively collected dataset of 191 breasts from 187 patients, the overall accuracies of the multimodal model and of preliminary pathologist-level assessments of biopsied breast specimens were similar (90.1% vs 92.7%, respectively). Multimodal models may assist diagnosis in oncology.