Incomplete Multimodal Federated Learning via Masking and Contrasting Prototypes

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Multimodal federated learning (mFL) faces challenges with random modality missingness in real-world applications.
• Existing mFL methods struggle with incomplete modalities, leading to performance degradation and task drift.

Purpose of the Study:

• To develop a novel mFL framework addressing task drift and performance loss due to missing modalities.
• To enhance generalization in complex, modality-missing scenarios during training and inference.

Main Methods:

• Utilized prototype learning to create a prototype library for enhancing FedAvg-based federated learning (FL).
• Employed prototypes as masks to compensate for missing modality information, formulating a task-calibrated training loss.
• Devised a model-agnostic strategy for modality-incomplete inference and integrated inter-client information via prototype contrastive learning.

Main Results:

• The proposed mFL framework demonstrated state-of-the-art performance across various missingness settings.
• Achieved significant improvements in inference performance compared to existing mFL methods under different missing modality rates.
• Showcased a 23.8% improvement in performance during modality-incomplete inference.

Conclusions:

• The novel mFL framework effectively alleviates task drift and performance degradation caused by modality missingness.
• The prototype-based approach offers a robust solution for real-world mFL challenges with incomplete data.
• The framework shows superior generalization and performance in complex, incomplete multimodal learning scenarios.