MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data

Hailong Yang ^1,2, Jia Wang ³, Wenyan Wang ¹

⁰School of Electrical and Information Engineering, Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma'anshan, Anhui 243032, China.

Briefings in Bioinformatics +

|

May 14, 2025

View abstract on PubMed

Summary

This study introduces MMSurv, a multimodal deep learning model for cancer survival prediction. By integrating clinical, sequencing, and imaging data, MMSurv significantly improves prediction accuracy compared to single-data sources.

Area Of Science

Oncology
Bioinformatics
Artificial Intelligence

Background

Accurate cancer patient survival prediction is crucial for treatment planning.
Current models often fail to leverage comprehensive multimodal data effectively.
This limits confidence in prognostic predictions.

Purpose Of The Study

To develop an interpretable multimodal deep learning model, MMSurv, for predicting cancer patient survival.
To integrate diverse data types including clinical information, sequencing data, and whole-slide images (WSIs).
To enhance predictive accuracy by effectively utilizing multimodal data complementarity.

Main Methods

MMSurv segments WSIs into tiles, encoding them into feature vectors using neural networks.
Clinical data is optimized using natural language processing-inspired word embedding techniques.
A novel fusion method combining compact bilinear pooling and Transformer architecture integrates multimodal features.
Dual-layer multi-instance learning refines predictions and enhances interpretability through cell segmentation analysis.

Main Results

Multimodal data integration improved predictive accuracy, increasing the C-index from 0.6750 to 0.7283 on average.
The proposed MMSurv model demonstrated a significant average performance improvement of nearly 10% over state-of-the-art methods.
Evaluation on six cancer types from The Cancer Genome Atlas confirmed the model's efficacy.

Conclusions

MMSurv offers a robust and interpretable approach to cancer survival prediction using multimodal data.
The model's ability to integrate diverse data sources significantly enhances prognostic accuracy.
This advancement holds promise for more personalized and effective cancer therapeutic planning.

Keywords:

C-index multi-cancer multi-instance learning survival prediction word embedding

Related Concept Videos

Cancer Survival Analysis

Cancer survival analysis focuses on quantifying and interpreting the time from a key starting point, such as diagnosis or the initiation of treatment, to a specific endpoint, such as remission or death. This analysis provides critical insights into treatment effectiveness and factors that influence patient outcomes, helping to shape clinical decisions and guide prognostic evaluations. A cornerstone of oncology research, survival analysis tackles the challenges of skewed, non-normally...

Combination Therapies and Personalized Medicine

Combining two or more treatment methods increases the life span of cancer patients while reducing damage to vital organs or tissue from the overuse of a single treatment. Combination therapy also targets different cancer-inducing pathways, thus reducing the chances of developing resistance to treatment.
The combination of the drug acetazolamide and sulforaphane is a good example of combination therapy to treat cancer. The cells in the interior of a large tumor often die due to the hypoxic and...