Monte Carlo Marginalization: A Differentiable Method to Learn High-Dimensional Distributions.

Area of Science:

• Machine Learning
• Computational Statistics
• Deep Generative Models

Background:

• Learning intractable distributions in high-dimensional spaces is a significant challenge.
• Current deep learning methods often impose restrictive prior assumptions.
• Efficiently approximating complex probability distributions is crucial for many AI applications.

Purpose of the Study:

• To propose a novel differentiable method for approximating intractable distributions.
• To shift the paradigm from network-dependent approximation to direct, network-free distribution learning.
• To address the computational complexity of Kullback-Leibler divergence in high-dimensional spaces.

Main Methods:

• Approximation of intractable distributions using a Gaussian mixture model (GMM).
• Minimization of Kullback-Leibler (KL) divergence via a novel Monte Carlo marginalization (MCMarg) method.
• Utilization of kernel density estimation (KDE) to ensure differentiability for intractable target distributions.

Main Results:

• Significant improvement in FID scores (approx. 10 points) when replacing standard priors in pretrained VAEs.
• Enables image generation without neural networks, achieving an FID of 22 on MNIST.
• Achieves an FID score of 2.69 on CIFAR-10, outperforming several state-of-the-art deep generative models.

Conclusions:

• The proposed MCMarg method offers a powerful and differentiable tool for learning complex distributions.
• This approach represents the first known attempt at image generation without relying on deep learning networks.
• The method demonstrates superior performance and opens new avenues for generative modeling.