Richer fusion network for breast cancer classification based on multimodal data

Area of Science:

• Medical Informatics
• Computational Pathology
• Artificial Intelligence in Medicine

Background:

• Single-mode pathological image classification for breast cancer lacks clinical accuracy.
• Pathologists integrate pathological images with electronic medical record (EMR) data for diagnosis.

Purpose of the Study:

• To improve breast cancer classification accuracy by integrating pathological images and structured EMR data.
• To develop a multimodal deep learning approach for enhanced breast cancer diagnosis.

Main Methods:

• Proposed a richer fusion network for multimodal breast cancer classification.
• Extracted multilevel pathological image features and used denoising autoencoders for EMR data.
• Enhanced low-dimensional EMR data to high-dimensional to minimize information loss before fusion.

Main Results:

• Achieved superior average classification accuracy of 92.9%, outperforming state-of-the-art methods.
• Released a publicly available dataset of 3764 breast cancer pathological images and 185 patients' EMR data.
• Demonstrated robustness to partially missing structured EMR data.

Conclusions:

• The richer fusion network effectively integrates heterogeneous data for improved pathological image classification.
• Enables practical application of automatic breast cancer classification in clinical settings.
• The fusion method is extensible to other structured and unstructured data types.