Factor-Assisted Federated Learning for Personalized Optimization With Heterogeneous Data

Federated learning is an emerging distributed machine learning framework aimed at protecting data privacy. Data heterogeneity is one of the core challenges in federated learning, which could severely degrade the convergence rate and prediction performance of deep neural networks. To address this issue, we develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit. This modeling framework is motivated by the finding that data on different clients contain both common knowledge and personalized knowledge. Then the hidden elements in each neural layer can be split into shared and personalized groups. With this decomposition, a novel objective function is established and optimized. We demonstrate that FedSplit enjoys a faster convergence speed than the standard federated learning method both theoretically and empirically. The generalization bound of the FedSplit method is also studied. To practically implement the proposed method on real datasets, factor analysis is introduced to facilitate the decoupling of hidden elements. This leads to a practically implemented model for FedSplit, which we further refer to as FedFac. We demonstrate by simulation studies that using factor analysis can well recover the underlying shared/personalized decomposition. The superior prediction performance of FedFac is further verified empirically by comparison with various state-of-the-art federated learning methods on several real datasets.