Ratio divergence learning using target energy in restricted Boltzmann machines: Beyond Kullback-Leibler divergence...

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Discrete energy-based models (EBMs) are powerful tools for modeling complex data distributions.
• Restricted Boltzmann machines (RBMs) are a fundamental discrete EBM satisfying the universal approximation theorem.
• Existing learning methods like forward and reverse Kullback-Leibler divergence (KLD) learning have limitations, including underfitting and mode collapse.

Purpose of the Study:

• To introduce Ratio Divergence (RD) learning, a new method for discrete energy-based models.
• To address the limitations of traditional KLD learning methods in discrete EBMs.
• To evaluate the effectiveness of RD learning compared to existing methods.

Main Methods:

• RD learning utilizes both training data and a tractable target energy function.
• RD learning combines the strengths of forward and reverse KLD learning.
• Numerical experiments were conducted using various discrete EBMs, including RBMs, with RD learning as the primary method.

Main Results:

• RD learning significantly outperforms existing methods in energy function fitting, mode-covering, and learning stability.
• The effectiveness of RD learning increases with the dimensionality of the target models.
• A baseline method summing forward and reverse KLD was evaluated against RD learning.

Conclusions:

• RD learning offers a robust and effective approach for training discrete energy-based models.
• RD learning successfully mitigates underfitting and mode collapse issues inherent in KLD learning.
• The proposed RD learning method shows promise for complex, high-dimensional discrete distribution modeling.