Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Cybersecurity

Background:

• Regular neural networks lack uncertainty quantification, a key advantage of Bayesian neural networks (BNNs).
• The integration of BNNs with differential privacy (DP) has remained largely unexplored, limiting privacy-aware uncertainty quantification.
• Existing DP methods often face a direct tradeoff between privacy guarantees and predictive accuracy.

Purpose of the Study:

• To bridge the gap between Bayesian deep learning and differential privacy by developing DP-BNNs.
• To precisely analyze the privacy-accuracy tradeoff in BNNs within the DP framework.
• To introduce novel DP-BNN methods for uncertainty quantification and evaluate their performance.

Main Methods:

• Proposed three distinct DP-BNN approaches: DP-SGLD (noisy gradient), DP-BBP (parameter perturbation), and DP-MC Dropout (architectural modification).
• Leveraged recent advancements in Bayesian deep learning and privacy accounting for precise analysis.
• Conducted extensive experiments to compare DP-BNNs against non-DP and non-Bayesian methods.

Main Results:

• Demonstrated an equivalence between DP-SGD and DP-SGLD, indicating inherent uncertainty quantification in some non-Bayesian DP training.
• Identified differing hyperparameter effects (learning rate, batch size) between DP-SGD and DP-SGLD.
• Observed a new privacy-reliability tradeoff; DP-SGLD achieved remarkable accuracy under strong privacy guarantees.

Conclusions:

• DP-BNNs successfully integrate uncertainty quantification with differential privacy.
• DP-SGLD presents a promising approach, offering high accuracy while maintaining strong privacy.
• DP-BNNs hold significant potential for real-world applications requiring both privacy and reliable predictions.