Multimodal breast cancer diagnosis using feature fusion and deep learning

Breast cancer is one of the leading health problems in the world, and the challenge lies in the fact that its diagnosis at the earliest possible and accurate rate is the main factor to guarantee a successful patient outcome. The traditional deep learning (DL) frameworks usually utilize data from a single modality at a time and, therefore, are not capable of addressing the complexity and heterogeneity of the disease, particularly when data are unavailable or incomplete.

To address these constraints, a multimodal breast cancer diagnosis model is presented that consists of attention-based transformers to achieve efficient modality specific feature extraction, the modified mantissa search (MMS) algorithm to remove irrelevant features, and the American zebra optimization (AZO) algorithm to dynamically and efficiently combine features. Final classification is then performed using a lightweight convolutional neural network (LCNN) to avoid compromising diagnostic accuracy.

The proposed model is highly generalizable and resilient to missing modalities, achieving 98.958, 97.37, and 99.438% accuracy on Mammographic Image Analysis Society (MIAS), BreakHis, and combined multimodal datasets, respectively. These findings reveal their usefulness and strength in clinical diagnostic cases with a variety of imaging data.