FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) enables collaborative model training across decentralized devices without sharing raw data.
• Federated distillation (FD) is an FL paradigm that distills client predictions on auxiliary data, allowing diverse model architectures.
• Existing FD methods may not fully leverage unlabeled auxiliary data for performance gains.

Purpose of the Study:

• To propose FedAUX, an extension of federated distillation that significantly improves performance by maximizing the utility of unlabeled auxiliary data.
• To introduce novel techniques within the FD framework to enhance distributed model training.
• To demonstrate substantial performance improvements over existing FL methods.

Main Methods:

• FedAUX incorporates unsupervised pre-training on auxiliary data for optimal model initialization.
• It employs (ε, δ)-differentially private certainty scoring to weight client model predictions based on confidence.
• The method is evaluated on large-scale convolutional neural networks (CNNs) and transformer models.

Main Results:

• FedAUX achieves remarkable performance improvements compared to state-of-the-art FL methods.
• Experiments show significant gains in validation accuracy, e.g., from 30.4% to 78.1% for ResNet8 on CIFAR10.
• The method demonstrates these improvements without substantial increases in computation, communication, or privacy overhead.

Conclusions:

• FedAUX effectively enhances federated distillation by better utilizing unlabeled auxiliary data.
• The proposed approach significantly closes the gap between federated and centralized training performance.
• FedAUX offers a practical and efficient method for improving federated learning outcomes.