ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders

Area of Science:

• Machine Learning
• Deep Learning
• Statistical Modeling

Background:

• Variational Autoencoders (VAEs) are popular Deep Latent-Variable Models (DLVMs) for data distribution modeling.
• Optimizing VAEs is more manageable than other DLVMs.
• The latent dimension size is a critical VAE hyperparameter, often determined empirically.

Purpose of the Study:

• To propose a statistical formulation for discovering relevant latent factors in datasets.
• To address the empirical trial-and-error approach for selecting VAE latent dimensions.
• To introduce the Automatic Relevancy Detection in Variational Autoencoder (ARD-VAE) method.

Main Methods:

• Implemented a hierarchical prior in the VAE's latent space.
• The hierarchical prior estimates latent axis variance using encoded data.
• Replaced the fixed prior in the VAE objective function with the hierarchical prior.

Main Results:

• The proposed ARD-VAE method effectively identifies relevant latent dimensions.
• Demonstrated efficacy on multiple benchmark datasets.
• Analyzed the impact of identified latent dimensions on metrics like FID score and disentanglement.

Conclusions:

• ARD-VAE offers a statistically grounded approach to determining VAE latent space dimensionality.
• The method automates the discovery of explanatory factors within datasets.
• ARD-VAE enhances VAE performance by optimizing latent representations.