Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

Area of Science:

• Biomedical Informatics
• Data Privacy
• Statistical Analysis

Background:

• Synthetic data generation is crucial for sharing sensitive biomedical information while preserving privacy.
• Differential Privacy (DP) is the standard for balancing data utility and individual privacy.
• Evaluating the trustworthiness of statistical findings from DP-synthetic data is essential.

Purpose of the Study:

• To assess the reliability of group difference discoveries from DP-synthetic data using independent sample tests.
• To quantify Type I (false discovery) and Type II (missed discovery) errors in statistical tests on DP-synthetic data.
• To understand the impact of different DP-synthetic data generation methods on statistical validity.

Main Methods:

• Evaluation of Mann-Whitney U, Student's t-test, chi-squared, and median tests.
• Generation of DP-synthetic data from real-world (prostate cancer, cardiovascular) and simulated datasets.
• Comparison of five DP-synthetic data generation methods, including histogram-based, MWEM, Private-PGM, and DP GAN.

Main Results:

• Many DP-synthetic data generation methods exhibited significantly inflated Type I errors at low privacy levels (epsilon <= 1).
• Low p-values in statistical tests may result from DP noise rather than true effects.
• A DP Smoothed Histogram method maintained valid Type I errors across privacy levels but required large datasets and epsilon >= 5 for acceptable Type II errors.

Conclusions:

• Caution is advised when interpreting statistical results from DP-synthetic data due to potential inflation of Type I errors.
• The choice of DP-synthetic data generation method critically impacts the validity and reliability of statistical analyses.
• Achieving both privacy and statistical accuracy requires careful consideration of dataset size, privacy budget, and generation methodology.