Multi-Modal Federated Learning for Cancer Staging Over Non-IID Datasets With Unbalanced Modalities.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computational Biology

Background:

• Machine learning (ML) and federated learning (FL) show promise for cancer staging via medical image analysis.
• Multi-modal learning frameworks integrating diverse patient data (e.g., images, genomics, clinical data) are crucial for accurate cancer staging.
• Existing multi-modal FL approaches often assume uniform data modality access across institutions, which is unrealistic.

Purpose of the Study:

• To develop a novel federated learning (FL) architecture for multi-modal cancer staging that accommodates data and modality heterogeneity across institutions.
• To address challenges in multi-modal FL, such as varying convergence speeds due to different data modalities.
• To propose a distributed gradient blending and proximity-aware client weighting strategy for robust multi-modal FL.

Main Methods:

• Introduced a novel FL architecture supporting non-uniform data modalities across participating institutions.
• Developed a distributed gradient blending and proximity-aware client weighting strategy to manage convergence heterogeneity.
• Conducted experiments using The Cancer Genome Atlas (TCGA) datalake with mRNA sequences, histopathology images, and clinical data.

Main Results:

• The proposed FL architecture effectively handles data and modality heterogeneity in multi-modal cancer staging.
• The gradient blending and client weighting strategy improved model convergence and performance in heterogeneous settings.
• Analysis revealed the distinct impacts of class-based versus type-based heterogeneity on model outcomes.

Conclusions:

• The novel FL architecture provides a more realistic and effective approach to multi-modal cancer staging.
• Addressing data modality non-uniformity is critical for successful federated multi-modal learning in healthcare.
• This work advances the understanding of data heterogeneity in multi-modal FL for cancer research.