The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions.

Area of Science:

• Machine Learning
• Computer Vision
• Probabilistic Modeling

Background:

• Restricted Boltzmann Machines (RBMs) are generative stochastic neural networks.
• Modeling natural images requires capturing complex statistical properties, including conditional covariance.
• Existing RBMs may not fully capture the nuances of high-dimensional data like natural images.

Purpose of the Study:

• To introduce the canonical spike-and-slab restricted Boltzmann machine (ssRBM) framework.
• To present extensions of the ssRBM, demonstrating its flexibility for advanced probabilistic models.
• To explore the application of ssRBMs and their extensions for learning invariant features in image data.

Main Methods:

• Definition of the ssRBM with real-valued 'slab' and binary 'spike' variables per hidden unit.
• Development of ssRBM extensions, including the subspace-ssRBM for invariant feature learning.
• Experimental evaluation on benchmark datasets: MNIST digit recognition and CIFAR-10 object classification.

Main Results:

• The ssRBM effectively models conditional covariance, crucial for natural image statistics.
• ssRBM extensions demonstrate adaptability for sophisticated probabilistic modeling of high-dimensional data.
• The subspace-ssRBM shows promise in learning invariant features for image recognition tasks.

Conclusions:

• The ssRBM framework offers a flexible platform for developing advanced probabilistic models.
• Extensions of the ssRBM enhance its capabilities for complex data, particularly natural images.
• The ssRBM approach shows potential for improved performance in image recognition and feature learning tasks.