Variational autoencoder for distributional learning via quantile function estimation

The Gaussianity assumption in Variational AutoEncoders (VAEs) enhances computational efficiency and provides a solid theoretical basis for estimating probability distributions. However, we have empirically found that approximating distributions with non-smooth densities using the Gaussian VAE is challenging. Therefore, we propose an approach for distributional learning in VAEs that extends to estimating the quantile function while accommodating both smooth and non-smooth densities. This is achieved by utilizing the continuous ranked probability score, a strictly proper scoring rule, as our reconstruction loss. Our method can be seen as a specialized form of a nonparametric M-estimator for estimating general quantile functions, and we establish a theoretical connection between our model and quantile estimation. Furthermore, we demonstrate that our reconstruction loss functions as the lower bound of an infinite mixture of asymmetric Laplace distributions, which allows our synthetic data generation mechanism to maintain differential privacy. We validate the effectiveness of our model in capturing the underlying distribution through experiments involving synthetic data generation on real-world tabular datasets, showing that the level of data privacy can be easily adjusted.