Clinical Stage Prompt Induced Multi-Modal Prognosis

Area of Science:

• Oncology
• Bioinformatics
• Medical Imaging

Background:

• Histology and genomic data are crucial for cancer prognosis.
• Integrating these modalities presents challenges in modeling high-order relevance and incorporating clinical knowledge.
• Existing methods often lack effective multi-modal fusion strategies for prognosis.

Purpose of the Study:

• To develop a novel framework for multi-modal cancer prognosis by integrating Whole Slide Images (WSIs) and genomic data.
• To address the challenges of high-dimensional data imbalance and the under-utilization of clinical knowledge in cancer survival prediction.
• To propose a prognosis-oriented image-omics fusion method that leverages clinical staging information.

Main Methods:

• Proposed Clinical Stage Prompt induced Multimodal Prognosis (CiMP) framework.
• Utilized advanced Large Language Models (LLMs) to generate clinical staging prompts from structured records.
• Introduced a Group Multi-Head Self-Attention module for genomic data analysis.
• Integrated WSIs and genomic data for multi-modal prognosis.

Main Results:

• Achieved state-of-the-art performance on five TCGA datasets.
• Demonstrated superiority over previous multi-modal prognostic models.
• Showcased improved clinical interpretability and potential for medical applications.

Conclusions:

• CiMP effectively integrates histology images and genomic data for enhanced cancer survival prediction.
• The framework successfully incorporates clinical knowledge through staging prompts, overcoming limitations of prior methods.
• The proposed approach holds significant promise for advancing personalized cancer medicine and clinical decision-making.