Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Privacy

Background:

• Deep learning models excel with large datasets but risk privacy leakage.
• Achieving a balance between high model utility and strong data privacy is crucial for practical applications.

Purpose of the Study:

• To propose a novel discriminative-generative distillation approach for learning privacy-preserving deep models.
• To enable effective knowledge transfer from private data to a student network while mitigating privacy risks.

Main Methods:

• A discriminative stream trains classifiers on private data and teacher ensembles.
• A generative stream, using a fixed discriminator, trains a generator to create synthetic data.
• Synthetic data trains a variational autoencoder (VAE), and labels are queried using differentially private aggregation.

Main Results:

• The approach generates synthetic data for privacy-preserving model training.
• It effectively transfers knowledge from teacher models to a student network.
• The method controls query costs and minimizes accuracy degradation.

Conclusions:

• The proposed discriminative-generative distillation effectively creates privacy-preserving deep models.
• This approach offers a viable solution for deploying deep learning while safeguarding sensitive data.
• Experimental results validate the effectiveness in achieving both privacy and utility.