FedMIR: Multimodal Federated Learning with Missing Modality Imputation and Distribution-Aware Routing.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Current multimodal federated learning methods fail with incomplete data and distribution variations.
• These limitations hinder applications in dynamic environments like the Internet of Things (IoT).
• Handling missing modalities and distribution drift is crucial for robust federated learning.

Purpose of the Study:

• To introduce FedMIR, a new framework for multimodal federated learning.
• To enable effective learning despite missing data modalities and distribution drift.
• To improve cross-modal understanding in decentralized learning systems.

Main Methods:

• Mapping heterogeneous modalities into a shared semantic space for dependency modeling.
• Utilizing contrastive learning for image-text alignment in a latent space.
• Employing conditional generation for reconstructing missing modality representations.
• Implementing a mixture-of-experts backbone conditioned on distribution state.
• Sharing only model parameters and distribution statistics to preserve privacy and enable adaptation.

Main Results:

• FedMIR effectively handles missing modalities during federated learning.
• The framework demonstrates adaptability to distribution drift using adaptive expert allocation.
• Evaluations on federated image-text retrieval benchmarks show superior performance over baselines under heterogeneity and missing data.

Conclusions:

• FedMIR offers a robust solution for multimodal federated learning in heterogeneous and incomplete data settings.
• The proposed approach enhances the applicability of federated learning in real-world distributed systems like IoT.
• Shared semantic spaces and adaptive routing are key to overcoming current federated learning challenges.