Multimodal Data Fusion for Whole-Slide Histopathology Image Classification

Area of Science:

• Computational pathology
• Artificial intelligence in oncology
• Digital pathology

Background:

• Whole slide images (WSIs) are crucial for cancer diagnosis but present computational challenges.
• Automated AI tools require precise annotations and extensive training data.
• Multimodal data integration (WSIs and pathology reports) can improve classification accuracy and reduce variability.

Purpose of the Study:

• To introduce MPath-Net, an end-to-end multimodal framework for enhanced cancer subtype classification.
• To leverage WSIs and pathology reports for improved diagnostic workflows.
• To demonstrate the clinical utility of multimodal AI in pathology.

Main Methods:

• Utilized the TCGA dataset (1684 cases: 916 kidney, 768 lung).
• Applied multiple-instance learning (MIL) for WSI feature extraction and Sentence-BERT for report encoding.
• Performed joint fine-tuning for tumor classification.

Main Results:

• MPath-Net achieved 94.65% accuracy, 0.9553 precision, 0.9472 recall, and 0.9473 F1-score.
• Significantly outperformed baseline models (P < 0.05).
• Attention heatmaps provided interpretable tumor tissue localization.

Conclusions:

• MPath-Net effectively integrates WSIs and pathology reports for accurate cancer subtype classification.
• The framework demonstrates significant potential to support pathologists, improve diagnostic accuracy, and reduce inter-reader variability.
• Findings advance precision medicine through multimodal AI integration in digital pathology.