Deep multimodal graph-based network for survival prediction from highly multiplexed images and patient variables

Area of Science:

• Oncology
• Computational Biology
• Medical Imaging

Background:

• Tumor microenvironment spatial architecture and cell heterogeneity impact cancer prognosis.
• High-dimensional imaging techniques like imaging mass cytometry (IMC) offer detailed spatial and biomarker data at single-cell resolution.
• Current survival prediction methods often fail to utilize comprehensive spatial phenotype information from IMC data.

Purpose of the Study:

• To develop an end-to-end method for enhanced cancer survival prediction.
• To integrate rich spatial phenotype information from whole IMC images with clinical data.
• To improve prognostic accuracy by leveraging multimodal data sources.

Main Methods:

• Introduction of a deep multimodal graph-based network (DMGN).
• A multimodal graph-based module adaptively considers spatial phenotypes across image regions and clinical variables.
• A clinical embedding module generates specialized embeddings for clinical variables to improve aggregation.

Main Results:

• The DMGN approach consistently improved survival prediction performance on two public breast cancer datasets.
• The proposed method demonstrated superior performance compared to existing state-of-the-art survival prediction techniques.
• Both modules of the DMGN contributed to enhanced survival prediction accuracy.

Conclusions:

• The deep multimodal graph-based network (DMGN) effectively integrates spatial imaging and clinical data for improved cancer survival prediction.
• This novel approach offers a significant advancement over current methods by leveraging single-cell spatial phenotypes.
• The DMGN has the potential to enhance patient-specific prognosis and clinical decision-making.