Deep Generative Models: Complexity, Dimensionality, and Approximation

Area of Science:

• Machine Learning
• Deep Learning
• Geometric Deep Learning

Background:

• Generative networks excel at learning complex data distributions.
• The theoretical basis for their success, especially regarding data dimensionality, is not fully understood.
• The manifold hypothesis suggests datasets have intrinsic low-dimensional structures.

Purpose of the Study:

• To investigate the theoretical limitations on latent dimension requirements for generative networks.
• To challenge the conventional belief that latent dimensions must match or exceed the manifold's dimension.
• To explore alternative approaches inspired by space-filling curves.

Main Methods:

• Theoretical analysis of generative networks.
• Application of space-filling curve concepts to neural network architecture.
• Derivation of complexity bounds for deep neural networks.

Main Results:

• Generative networks can approximate distributions on d-dimensional Riemannian manifolds using arbitrary input dimensions, including dimensions lower than d.
• This capability is achieved by adapting network structures inspired by space-filling curves.
• A super-exponential complexity bound for deep neural networks was established.

Conclusions:

• The requirement for latent dimension to be at least d or d+1 is not necessary for approximating distributions on d-dimensional manifolds.
• Findings challenge established notions regarding input dimensionality and generative network capabilities.
• Highlights a trade-off between approximation error, dimensionality, and model complexity in generative models.