Colorectal Cancer Survival Prediction Using Multimodal Fusion.

Area of Science:

• Oncology
• Medical Informatics
• Computational Biology

Background:

• Colorectal cancer (CRC) survival prediction remains challenging.
• Integrating diverse data types like whole-slide images (WSIs) and clinical data can improve prognostic accuracy.
• Existing models often analyze these data modalities separately.

Purpose of the Study:

• To develop and evaluate a novel multimodal fusion framework for colorectal cancer survival prediction.
• To integrate WSIs and tabular clinical/biomarker data for enhanced prognostic modeling.
• To compare different fusion strategies, including contrastive learning and cross-attention.

Main Methods:

• A multimodal fusion framework was designed, utilizing ProvGigaPath for WSI feature extraction and TabTransformer for tabular data.
• Two fusion strategies were investigated: contrastive learning (CL) for unimodal analysis and cross-attention (CA) for multimodal integration.
• Experiments were conducted on the Molecular and Cellular Oncology (MCO) dataset.

Main Results:

• Contrastive learning achieved an area under the receiver operating characteristic curve (AUROC) of 0.82 for tabular data and 0.73 for WSI data.
• The cross-attention-based multimodal fusion model achieved a C-index of 0.77.
• The multimodal fusion approach demonstrated superior performance compared to unimodal baseline models.

Conclusions:

• Multimodal fusion of WSI and clinical data offers a promising approach for improving colorectal cancer survival prediction.
• Cross-attention is an effective strategy for integrating intermediate-level features from different data modalities.
• The proposed framework has the potential to enhance clinical decision-making for CRC patients.