Optimizing Unnormalized Statistical Models Through Compositional Optimization.

Area of Science:

• Machine Learning
• Statistical Modeling
• Optimization Theory

Background:

• Learning unnormalized statistical models, such as energy-based models, is computationally intensive due to the difficulty of handling the partition function.
• Noise-contrastive estimation (NCE) simplifies this by using logistic loss but often suffers from flat loss landscapes and slow convergence.

Purpose of the Study:

• To develop a direct and efficient method for optimizing the negative log-likelihood of unnormalized models.
• To address the limitations of NCE by proposing a novel approach based on compositional optimization.

Main Methods:

• Introduced a noise distribution to express the log partition function as a compositional function, enabling estimation via stochastic samples.
• Applied stochastic compositional optimization algorithms to directly optimize the model's objective function.
• Analyzed convergence rates and dependence on noise distribution properties.

Main Results:

• Established a fast convergence rate for the proposed method, quantifying its dependence on the noise distribution's variance.
• Demonstrated a more favorable loss landscape and faster convergence compared to NCE in Gaussian mean estimation.
• Achieved superior performance in density estimation, out-of-distribution detection, and real image generation tasks.

Conclusions:

• The proposed direct optimization approach via stochastic compositional optimization is more favorable than NCE for unnormalized models.
• This method offers significant improvements in convergence speed, loss landscape, and performance across various machine learning applications.