Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable...

Area of Science:

• Computational biology
• Bioinformatics
• Medical image analysis

Background:

• Multimodal data integration, combining gene expression and pathological images, offers superior cancer survival analysis compared to single-modality approaches.
• Existing methods often overlook detailed interactions between modalities, particularly between biological pathways and specific image regions.

Purpose of the Study:

• To propose a novel Pathway-Aware Multimodal Transformer (PAMT) framework for interpretable cancer survival analysis.
• To enhance the understanding of fine-grained interactions between biological pathways and pathological image patches.

Main Methods:

• The PAMT framework employs a three-stage process: intra-modal interaction using Transformers, label-free contrastive loss for inter-modal alignment, and a pathway-to-patch cross-fusion module guided by pathway priors.
• This approach models the genotype-phenotype relationship by integrating pathway information with image features.

Main Results:

• The PAMT framework demonstrated significant performance improvements over state-of-the-art methods on three cancer datasets (bladder urothelial carcinoma, lung squamous cell carcinoma, lung adenocarcinoma).
• The cross-fusion module provides interpretability, enabling identification of key pathways and affected image regions.

Conclusions:

• The proposed PAMT framework effectively integrates multimodal data for improved cancer survival prediction.
• The interpretability of PAMT aids pathologists in understanding pathway-specific impacts on whole slide images (WSIs) and identifying prognostic patterns.