A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction.

Area of Science:

• Oncology
• Bioinformatics
• Computational Pathology

Background:

• Histopathological images and genomic data offer complementary insights into cancer mechanisms.
• Current multi-modal cancer research often focuses on single tasks, overlooking inter-task correlations.
• Integrating diverse data types is crucial for advancing cancer diagnosis, prognosis, and therapy.

Purpose of the Study:

• To develop a multi-modal fusion framework, MultiCoFusion, that leverages multi-task correlation learning.
• To simultaneously address cancer survival analysis and grade classification by integrating imaging and genomic data.
• To improve the understanding of complex cancer mechanisms through integrated data analysis.

Main Methods:

• Utilized a pre-trained ResNet-152 for histopathological image representation learning.
• Employed a sparse graph convolutional network (SGCN) for mRNA expression data representation learning.
• Integrated representations using a multi-task shared fully connected neural network (FCNN) within an alternating training scheme.

Main Results:

• MultiCoFusion demonstrated superior representation learning compared to traditional feature extraction methods.
• The framework achieved enhanced performance in both survival analysis and cancer grade classification.
• Multi-task alternating learning improved performance across multiple tasks, outperforming existing deep learning and traditional methods.

Conclusions:

• MultiCoFusion effectively integrates multi-modal data for enhanced cancer research.
• Multi-task learning significantly boosts performance in both single-modal and multi-modal cancer analyses.
• The framework provides a robust approach for simultaneous survival analysis and grade classification in cancers.