Monte Carlo Marginalization: A Differentiable Method to Learn High-Dimensional Distributions

Learning intractable distributions in high-dimensional spaces remains a fundamental challenge. While prevalent deep learning methods often rely on restrictive prior assumptions, we propose a novel differentiable method that approximates intractable distributions using a Gaussian mixture model (GMM) by minimizing Kullback-Leibler (KL) divergence. In particular, a novel Monte Carlo marginalization (MCMarg) method is proposed to address the computational complexity of the KL divergence, which is unacceptable in a high-dimensional space. In addition, kernel density estimation (KDE) is utilized to ensure the differentiability of the optimization process because the target distribution is intractable. The proposed approach is a powerful and differentiable tool for learning complex distributions, which shifts the paradigm from network-dependent approximation to direct, network-free distribution learning. Comprehensive experiments demonstrate the superior properties of the proposed approach. By replacing standard priors in pretrained VAEs, our method achieves a significant improvement of approximately 10 points in FID scores. Remarkably, the model enables image generation without using a neural network, achieving an FID of 22 on the MNIST dataset. On the CIFAR-10 benchmark, our method achieves an FID score of 2.69, outperforming several state-of-the-art deep generative models. To the best of our knowledge, the proposed MCMarg is the first attempt at image generation without using a deep learning network.