An integrated evaluation framework for synthetic clinical data in severely imbalanced settings: fidelity,...

The development of clinical artificial intelligence models is constrained by limited access to high-quality electronic health record data, a challenge that is particularly pronounced in rare and highly imbalanced clinical cohorts. Synthetic data generation has been proposed as a strategy to mitigate data-sharing barriers. However, an integrated evaluation framework that jointly examines distributional fidelity, diagnostic behavior, and privacy risk under such conditions remains lacking.

We developed an integrated evaluation framework to assess variational autoencoders (VAE) and conditional generative adversarial networks (CTGAN). The framework jointly characterizes distributional fidelity, privacy risk, and diagnostic behavior using structured electronic health record data from the KoGES cohort, with disease prevalence ranging from 0.8% to 7.5%, adopting a prevalence-aware approach in which evaluation metrics are stratified by disease-specific class frequency. To address the sensitivity of p-value-based tests in large samples, distance-based metrics, including Jensen-Shannon divergence and Wasserstein distance, were employed. Diagnostic behavior was evaluated using XGBoost, random forest, and logistic regression classifiers, with emphasis on minority-class-sensitive metrics such as recall and Macro-F1.

In multivariate structural analyses, the correlation similarity between empirical and synthetic data was 0.794 for VAE-generated data and 0.667 for CTGAN-generated data. Across diseases with moderate outcome prevalence, multivariate and stratified distributions exhibited numerical overlap between VAE-generated and empirical data. In diagnostic evaluations, classifiers trained on empirical data alone yielded zero recall for the rarest outcome (0.8% prevalence), whereas CTGAN-trained classifiers produced non-zero recall values at the cost of reduced overall accuracy. Across evaluated threat models, membership inference attack performance remained near the random-guessing reference (AUROC ≈ 0.500).

This study presents an integrated, prevalence-aware evaluation framework for synthetic clinical data that systematically identified failure modes undetectable by conventional single-metric approaches, including minority-class metric instability, heterogeneous tail-end privacy exposure, and qualitative divergence in membership score calibration. The evaluated generative models exhibited distinct trade-off profiles: VAE preserved multivariate structure while exhibiting lower proximity-based exposure under the evaluated threat model, whereas CTGAN achieved higher minority-class detection at the cost of structural fidelity. Supplementary augmentation experiments confirmed that increased synthetic data volume does not uniformly improve minority-class detection under extreme prevalence constraints. These findings demonstrate that numerical overlap between synthetic and empirical distributions does not guarantee clinical equivalence, underscoring the need for prevalence-stratified, multi-dimensional validation in structured EHR research.

Not applicable.