MPT-MIL: Multimodal Aware Prompt Tuning for Prediction of Cancer Survival

Area of Science:

• Oncology
• Computational Pathology
• Bioinformatics

Background:

• Survival prediction is crucial in oncology, utilizing pathology and genomic data.
• Current methods struggle with adapting pre-trained vision models for whole slide image (WSI) analysis and integrating gene expression data.
• Challenges include suboptimal feature extraction and failure to incorporate repetitive gene information.

Purpose of the Study:

• To develop an improved foundation model tuning strategy for oncology survival prediction.
• To enhance the integration of pathology (WSI) and genomic data.
• To overcome limitations of existing vision foundation model adaptation and gene expression data integration.

Main Methods:

• Proposed a plug-and-play multiple instance learning (MIL)-based foundation model tuning strategy.
• Introduced Task-specific Instance Selection using zero-shot learning for efficient WSI region selection.
• Developed a multi-model prompt token for integrating genetic information during prompt-tuning.
• Implemented a Gene Distribution Aware Task as an auxiliary task to improve multimodal learning.

Main Results:

• The proposed model demonstrated superior performance on three public TCGA datasets.
• Outperformed all previous MIL-based methodologies and fine-tuning approaches.
• Task-specific Instance Selection improved tuning efficiency and reduced irrelevant data interference.
• The auxiliary task enhanced the model's perception of multimodal information.

Conclusions:

• The novel MIL-based foundation model tuning strategy effectively integrates WSI and genomic data for survival prediction.
• The proposed methods, including Task-specific Instance Selection and the Gene Distribution Aware Task, significantly enhance model performance.
• This approach represents a substantial advancement in computational pathology for oncology research.