Heterogeneity-Aware Personalized Federated Neural Architecture Search

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) enables collaborative model training across decentralized devices.
• Resource and statistical heterogeneity in FL pose significant challenges to model training and personalization.
• Neural Architecture Search (NAS) can optimize models but struggles with FL's heterogeneity.

Purpose of the Study:

• To propose a novel method, Heterogeneity-Aware Personalized Federated NAS (HAPFNAS), to address challenges in FL.
• To enable efficient and effective personalized model discovery in heterogeneous federated environments.
• To improve the stability and performance of one-shot NAS within federated settings.

Main Methods:

• Implemented knowledge distillation from clients to a server-side supernet using lightweight models.
• Developed random-forest-based predictors for efficient, personalized architecture performance evaluation.
• Introduced heteroFedAvg, a model-heterogeneous FL algorithm for collaborative training of personalized models.

Main Results:

• HAPFNAS effectively mitigates heterogeneity issues in federated NAS.
• The proposed method enhances the stability of supernet training and the accuracy of architecture evaluation.
• Experimental results on CIFAR-10/100 and Tiny-ImageNet demonstrate superior performance compared to existing federated NAS approaches.

Conclusions:

• HAPFNAS successfully enables personalized model discovery in heterogeneous federated learning scenarios.
• The integration of knowledge distillation and personalized predictors is key to overcoming FL heterogeneity.
• This work advances the field of federated NAS by providing a robust and effective solution.