Restricted Boltzmann Machines as Models of Interacting Variables

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Restricted Boltzmann Machines (RBMs) are generative stochastic neural networks used for unsupervised learning.
• The choice of activation function in hidden nodes significantly impacts RBMs' representational capacity.
• Understanding these distributions is crucial for effective application in various domains.

Purpose of the Study:

• To investigate the precise distributions modeled by Restricted Boltzmann Machines (RBMs) with varying hidden node activation functions.
• To analyze the effect of activation functions on the marginal distributions of observed binary nodes.
• To determine how activation functions and model complexity affect RBMs' approximation accuracy.

Main Methods:

• Deriving an exact expression for marginal distributions imposed by RBMs on observed binary nodes.
• Analyzing the properties of interaction terms within the derived model, explicitly dependent on activation functions.
• Evaluating RBM approximation accuracy across different activation functions and numbers of hidden nodes.
• Testing the weak parameter approximation on RBMs trained with MNIST data.

Main Results:

• An exact expression for marginal distributions was found, revealing interactions dependent on the hidden node activation function.
• In the weak parameter regime, interaction terms showed reduced differences across activation functions, revealing an intuitive pattern.
• The weak parameter approximation proved effective for RBMs trained on the MNIST dataset.
• Inferred models were characterized as essentially low-order interaction models under the weak parameter approximation.

Conclusions:

• The activation function of hidden nodes in RBMs critically shapes the interactions and thus the expressible marginal distributions.
• The weak parameter approximation offers a simplified yet accurate view of RBM behavior, particularly for models trained on real-world data like MNIST.
• RBMs, especially under weak parameter conditions, tend to learn models with inherent low-order interactions, irrespective of the specific activation function used.