Multimodal Co-Attention Fusion Network With Online Data Augmentation for Cancer Subtype Classification

Area of Science:

• Computational pathology
• Bioinformatics
• Artificial intelligence in medicine

Background:

• Accurate cancer subtype diagnosis is crucial for personalized cancer treatment.
• Combining whole slide images (WSIs) and multi-omics data shows promise for improved diagnostic accuracy.
• Challenges include multimodal data heterogeneity and performance degradation due to limited patient data.

Purpose of the Study:

• To propose a novel multimodal co-attention fusion network (MCFN) with online data augmentation (ODA) for cancer subtype classification.
• To address data heterogeneity and the small-sample size problem in multimodal cancer data analysis.
• To enhance the accuracy and robustness of computational pathology diagnostics.

Main Methods:

• Developed a multimodal mutual-guided co-attention (MMC) module for dense multimodal interactions and heterogeneity alleviation.
• Introduced a self-normalizing network (SNN)-Mixer to facilitate information exchange among omics data and address high-dimensional small-sample issues.
• Implemented an online data augmentation (ODA) module to guide WSI augmentation using multimodal knowledge, compensating for limited data.

Main Results:

• The proposed MCFN demonstrated superior performance compared to existing algorithms on the TCGA dataset.
• The MMC module effectively reduced inter- and intra-modal data heterogeneities.
• The ODA module successfully maximized data diversity, improving model training with limited samples.

Conclusions:

• The novel MCFN with MMC and ODA modules significantly enhances cancer subtype classification accuracy.
• This approach effectively tackles multimodal data heterogeneity and data scarcity challenges in computational pathology.
• The findings support the potential of integrated multimodal data analysis for advancing personalized cancer treatment.