Heterogeneous Multi-Party Learning With Data-Driven Network Sampling.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Distributed Computing

Background:

• Multi-party learning trains models on decentralized data from multiple participants.
• Heterogeneous, non-Independent and Identically Distributed (non-IID) data across participants is a major challenge.
• Existing methods struggle with data heterogeneity in decentralized learning.

Purpose of the Study:

• To propose a novel framework, heterogeneous differentiable sampling (HDS), to address non-IID data challenges in multi-party learning.
• To enable local participants to extract optimal, smaller local models adapted to their data.
• To improve the performance and convergence speed of the global model.

Main Methods:

• Developed a heterogeneous differentiable sampling (HDS) framework.
• Incorporated a data-driven network sampling strategy inspired by dropout.
• Utilized differentiable sampling rates for adaptive local model extraction.
• Facilitated co-adaptation between local and global models.

Main Results:

• The HDS framework allows local participants to extract optimal, size-reduced local models.
• Achieved improved learning performance for the global model under non-IID data distributions.
• Demonstrated accelerated convergence of the global model.
• Outperformed several popular multi-party learning techniques in experiments.

Conclusions:

• The proposed HDS framework effectively handles non-IID data in multi-party learning.
• HDS offers a scalable and efficient approach for decentralized model training.
• This method enhances both local model efficiency and global model performance.