Foundation Model-Enabled Multimodal Deep Learning for Prognostic Prediction in Colorectal Cancer with Incomplete...

Area of Science:

• Oncology
• Artificial Intelligence
• Medical Imaging

Background:

• Accurate colorectal cancer prognosis is crucial for personalized treatment and improved patient outcomes.
• Current methods struggle with multimodal data integration and missing data.
• There is a need for advanced AI tools to enhance cancer prognosis prediction.

Purpose of the Study:

• To introduce FLARE, a novel multimodal deep learning framework for colorectal cancer prognosis.
• To leverage pathological images, radiological imaging, and clinical text reports for risk assessment.
• To address and mitigate performance issues caused by missing data modalities.

Main Methods:

• FLARE utilizes foundation models for efficient feature extraction.
• An attention-based multi-branch framework enhances inter-modal synergy and distinctiveness.
• Modality and missing-aware prompts, pseudo embeddings, and augmentation strategies handle incomplete data.

Main Results:

• FLARE demonstrated superior prognostic capability in a cohort of 1679 patients.
• It achieved the highest concordance index, outperforming existing models.
• The framework effectively stratified patients into high- and low-risk groups with robust generalizability.

Conclusions:

• FLARE offers a robust framework for multimodal deep learning in medical prognostics.
• It provides an advanced AI tool for precision cancer prognosis and intelligent diagnosis.
• The framework shows significant potential for optimizing colorectal cancer treatment strategies.