Deep learning with multimodal representation for pancancer prognosis prediction.

Area of Science:

• Oncology
• Bioinformatics
• Artificial Intelligence

Background:

• Accurate cancer patient prognosis is crucial for treatment planning.
• Current methods struggle to integrate diverse patient data, including clinical, genomic, and histopathology information.
• Multimodal data offers a richer source for improving prognostic accuracy.

Purpose of the Study:

• To develop a multimodal neural network model for predicting overall survival in 20 different cancer types.
• To effectively integrate and analyze multimodal patient data, including clinical information, mRNA expression, microRNA expression, and whole slide images (WSIs).
• To create a flexible and robust model capable of handling missing data and providing informative patient representations.

Main Methods:

• A multimodal neural network was constructed, incorporating an unsupervised encoder to generate a unified feature vector from four data modalities.
• Deep highway networks were used for clinical and genomic data, while convolutional neural networks processed WSIs.
• A resilient multimodal dropout method was employed to manage missing data during training.

Main Results:

• The pancancer model achieved an overall C-index of 0.78 in predicting overall survival.
• The model demonstrated the ability to predict prognosis for both individual cancer types and across multiple cancers simultaneously.
• The developed approach efficiently analyzes WSIs and represents multimodal patient data in an unsupervised manner.

Conclusions:

• A powerful automated tool for accurate cancer prognosis has been developed.
• The multimodal model successfully integrates diverse data types for improved survival prediction.
• This work represents a significant step towards personalized cancer treatment strategies.