Optimizing connectivity through network gradients for Restricted Boltzmann Machines.

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Sparse network connectivity enhances deep neural networks.
• Network connectivity is crucial for shallow network learning, like Restricted Boltzmann Machines (RBMs).
• Optimizing sparse connectivity in RBMs is challenging, often relying on hyperparameters.

Purpose of the Study:

• To introduce Network Connectivity Gradients (NCG), a novel optimization method for RBMs.
• To enable joint learning of RBM parameters and network connectivity without altering the energy-based objective.
• To efficiently discover optimal sparse connectivity patterns for improved RBM performance.

Main Methods:

• NCG utilizes network gradients to determine the influence of each connection.
• A continuous connection strength parameter is driven by gradients to define the connectivity pattern.
• RBM parameters and network connections are learned jointly, with distinct learning rates.

Main Results:

• NCG was applied to MNIST and other datasets, yielding improved RBM models.
• The method demonstrated superior performance in benchmark tasks of sample generation and classification.
• NCG proved robust to network initialization and capable of dynamically adding/removing connections.

Conclusions:

• NCG offers an effective approach for optimizing sparse connectivity in RBMs.
• The method achieves superior performance in generative and discriminative tasks.
• NCG provides a principled and flexible way to learn network structures in RBMs.