FedMEKT: Distillation-based embedding knowledge transfer for multimodal federated learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Federated learning (FL) traditionally focuses on unimodal data.
• Existing FL systems often require labeled client-side data, limiting real-world applicability.
• Exploiting multimodal data in FL is crucial for personalized applications.

Purpose of the Study:

• To introduce FedMEKT, a novel multimodal federated learning framework.
• To address challenges of modality discrepancy and limited labeled data in FL.
• To leverage semi-supervised learning for multimodal data representation.

Main Methods:

• Developed FedMEKT framework with local multimodal autoencoder learning, generalized multimodal autoencoder construction, and generalized classifier learning.
• Implemented a distillation-based multimodal embedding knowledge transfer mechanism for server-client data exchange.
• Utilized upstream and downstream multimodal embedding knowledge transfer for iterative global encoder updates.

Main Results:

• FedMEKT demonstrated superior global encoder performance in linear evaluation across four multimodal datasets.
• The framework ensures user privacy for personal data and model parameters.
• Achieved lower communication costs compared to existing baseline methods.

Conclusions:

• FedMEKT effectively enables multimodal federated learning using semi-supervised approaches.
• The proposed framework overcomes limitations of unimodal FL and labeled data dependency.
• FedMEKT offers a privacy-preserving, efficient solution for multimodal data analysis in decentralized settings.