A(DP) ²SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent With Differential Privacy

Area of Science:

• Machine Learning
• Distributed Systems
• Data Privacy

Background:

• Deep learning models require distributed learning for efficiency and privacy, especially in sensitive areas like healthcare.
• Asynchronous decentralized parallel SGD (ADPSGD) offers efficient, server-less distributed training but faces privacy risks from information leakage during communication.
• Existing methods lack robust privacy guarantees against malicious participants in decentralized settings.

Purpose of the Study:

• To develop a differentially private version of the asynchronous decentralized parallel SGD framework.
• To maintain the communication efficiency of ADPSGD while preventing information inference by malicious participants.
• To provide theoretical and empirical validation of the proposed privacy-preserving distributed learning method.

Main Methods:

• Introduction of the A(DP)2SGD framework, a novel differentially private asynchronous decentralized parallel SGD.
• Utilization of Rényi differential privacy for tighter privacy analysis with composite Gaussian mechanisms.
• Theoretical convergence analysis demonstrating an optimal O(1/√T) rate, consistent with standard SGD.

Main Results:

• A(DP)2SGD effectively prevents information inference by malicious participants in decentralized learning.
• The framework achieves a convergence rate consistent with its non-private counterpart.
• Empirical results show comparable model accuracy to differentially private Synchronous SGD (SSGD) but significantly faster training times in heterogeneous environments.

Conclusions:

• A(DP)2SGD successfully integrates differential privacy into asynchronous decentralized parallel SGD, enhancing security in distributed deep learning.
• The proposed method offers a practical and efficient solution for privacy-preserving distributed training, outperforming synchronous methods in speed.
• This work advances the field of secure and efficient distributed machine learning, particularly for applications requiring stringent data privacy.