Evaluating the quality of tabular synthetic data in health care

Machine Learning (ML) research in healthcare remains challenging as large, privacy-preserving open datasets are lacking. Synthetic data could offer a solution, but the value of synthetic data depends on diverse and conflicting criteria such as utility, fidelity, and privacy, which are rarely evaluated comprehensively. To close this gap, we explore the trade-off between these metrics in an empirical evaluation across a broad spectrum of generative models, datasets and metrics. In order to include as many metrics and models as possible and to ensure both applicability and comparability with other studies, we focus on the most widely available data modality and task setting: tabular data associated with a classification task. Extending prior work our results demonstrate that no single generative model excels across all metrics and datasets. Across 9 datasets and 11 generative models, the first principal variance direction of all metrics captures the dominant trade-off between fidelity and utility metrics on one side and the privacy metrics on the other side. Sensitivity analyses indicate that the privacy-fidelity/utility trade-off captured by the first principal variance direction remains consistent across several datasets and may support model selection. These insights highlight the potential of synthetic data for responsible data sharing in health care as well as the need for better tooling in synthetic data generation with a higher degree of automation when optimizing for metrics capturing fidelity, utility and privacy.