Flexible Vertical Federated Learning With Heterogeneous Parties

Area of Science:

• Distributed machine learning
• Federated learning algorithms

Background:

• Vertical federated learning (VFL) trains models on data partitioned vertically across multiple parties.
• Existing VFL methods struggle with heterogeneous systems where party characteristics (speed, architecture, optimizers) vary and change over time.

Purpose of the Study:

• To introduce Flexible Vertical Federated Learning (Flex-VFL), a novel distributed algorithm for training smooth, non-convex functions.
• To address the challenges of heterogeneity and dynamic changes in distributed systems with vertically partitioned data.

Main Methods:

• Flex-VFL employs parallel block coordinate descent (P-BCD), where each party trains a model partition using stochastic coordinate descent.
• Theoretical convergence analysis is provided, demonstrating that convergence rate depends on party speeds and local optimizer parameters.

Main Results:

• The convergence rate of Flex-VFL is theoretically shown to be constrained by party speeds and local optimizer parameters.
• An adaptive extension to Flex-VFL is developed to adjust learning rates dynamically based on changing speeds and optimizer parameters.

Conclusions:

• Flex-VFL offers an effective solution for distributed machine learning in heterogeneous environments.
• The adaptive extension of Flex-VFL enhances its robustness and performance in dynamic systems, outperforming synchronous and asynchronous VFL algorithms in convergence time.