Masked Autoencoder for Distribution Estimation on Small Structured Data Sets.

Area of Science:

• Machine Learning
• Statistical Modeling
• Artificial Intelligence

Background:

• Autoregressive models are powerful for distribution estimation but require large datasets.
• Existing models like Masked Autoencoder for Distribution Estimation (MADE) do not utilize known data structural information.
• This limitation is particularly problematic when training on small datasets.

Purpose of the Study:

• To develop improved density estimation methods for small datasets with known structural information.
• To enhance the performance of autoregressive models by incorporating prior knowledge of data dependencies.
• To adapt the MADE architecture to leverage Markov Random Field (MRF) structures.

Main Methods:

• Proposed two novel autoencoder architectures for density estimation.
• Modified the masking process of MADE based on conditional dependencies derived from MRF structures.
• Reduced model or problem complexity by incorporating known data structures.

Main Results:

• The proposed methods demonstrated improved density estimation on small datasets.
• Evaluated performance against existing binary, discrete, and continuous density estimators.
• Tested on benchmark datasets including MNIST, binarized MNIST, OCR-letters, and synthetic data.

Conclusions:

• Incorporating known data structures, such as MRFs, significantly improves autoregressive density estimation on small datasets.
• The modified MADE architectures offer a promising approach for efficient learning with limited data.
• The proposed methods provide a flexible framework for leveraging structural priors in neural density estimation.