Scaling up spike-and-slab models for unsupervised feature learning.

Area of Science:

• Machine Learning
• Computer Vision

Background:

• Spike-and-slab models are effective for real-valued data modeling.
• Object recognition tasks benefit from advanced feature extraction methods.

Purpose of the Study:

• To introduce a faster approximate inference algorithm for spike-and-slab sparse coding (S3C).
• To develop and evaluate a deep variant of S3C, the partially directed deep Boltzmann machine (PD-DBM).
• To demonstrate the efficacy of these models in object recognition tasks.

Main Methods:

• Developed a faster approximate inference algorithm for S3C.
• Introduced and implemented a partially directed deep Boltzmann machine (PD-DBM).
• Extended the S3C inference algorithm for the PD-DBM.
• Described learning procedures for both models.

Main Results:

• The improved S3C inference algorithm allows for scaling to large problem sizes.
• S3C as a feature extractor yields strong object recognition performance, particularly with limited labeled data.
• The PD-DBM generates superior samples compared to its shallow counterpart.
• PD-DBM training is successful without greedy layerwise pre-training, unlike standard DBMs/DBNs.

Conclusions:

• The enhanced S3C inference algorithm significantly improves scalability for real-valued data modeling.
• S3C and PD-DBM offer powerful tools for object recognition, especially in low-data regimes.
• The PD-DBM presents a novel deep learning architecture trainable end-to-end, overcoming limitations of existing deep Boltzmann machines.