Tackling Modality-Heterogeneous Client Drift Holistically for Heterogeneous Multimodal Federated Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Medical Imaging

Background:

• Multimodal Federated Learning (MFL) enables collaborative model training across decentralized devices, respecting data privacy.
• A shift from homogeneous MFL (uniform modalities) to heterogeneous MFL (diverse modalities) is occurring, reflecting real-world scenarios like varying medical imaging availability (e.g., MRI, CT).
• Heterogeneous MFL faces a challenge known as modality-heterogeneous client drift, caused by differing local optimization due to unique data modalities.

Purpose of the Study:

• To introduce FedMM, a novel approach designed to mitigate modality-heterogeneous client drift in MFL.
• To enhance the convergence and performance of MFL models in scenarios with varying data modalities across clients.

Main Methods:

• FedMM employs modality dropout during local optimization, randomly masking modalities to encourage weight alignment while maintaining model expressivity.
• A task-specific inter- and intra-modal regularizer is incorporated to further stabilize weight distribution across different modalities, aiding the modality dropout process.
• The combined techniques holistically address client drift by promoting convergence among client models despite differing input modalities.

Main Results:

• FedMM effectively addresses client drift in heterogeneous MFL settings.
• The approach fosters convergence among client models, even with unique input modalities.
• Comprehensive evaluations on three medical image segmentation datasets demonstrated FedMM's superior performance compared to existing state-of-the-art heterogeneous MFL methods.

Conclusions:

• FedMM offers a simple yet effective solution for modality-heterogeneous client drift in MFL.
• The method enhances collaborative learning performance in diverse, real-world MFL applications, particularly in medical imaging.
• FedMM represents a significant advancement in heterogeneous MFL, enabling more robust and adaptable decentralized learning systems.