Adaptive Guidance for Local Training in Heterogeneous Federated Learning

Model heterogeneity poses a significant challenge in Heterogeneous Federated Learning (HtFL). In scenarios with diverse model architectures, directly aggregating model parameters is impractical, leading HtFL methods to incorporate an extra guiding objective alongside the original local objective on each client to facilitate collaboration. However, this often results in a mismatch between the guiding and local objectives. To resolve this mismatch and ensure the guiding objective consistently benefits each client's original objective rather than interfering with it, we propose Federated Learning-to-Guide (FedL2G), a federated method that adaptively learns to guide local training in a meta-learning manner. In this way, the client can learn the necessary knowledge from the process of aligning globally shared information, thereby compensating for the required information for its local task. We theoretically prove that FedL2G achieves a non-convex convergence rate of $\mathcal {O}(1/T)$. We also conduct extensive experiments across three types of data heterogeneity and six model heterogeneity settings, using 14 heterogeneous model architectures (e.g., CNNs and ViTs). The results show that FedL2G consistently outperforms 12 state-of-the-art methods in terms of test accuracy (both in-distribution and out-of-distribution) as well as communication and computational overhead, across standard benchmark datasets and specialized medical scenarios.