Justifying and generalizing contrastive divergence.

Area of Science:

• Machine Learning
• Statistical Modeling
• Artificial Intelligence

Background:

• Undirected graphical models, such as Restricted Boltzmann Machines (RBMs), are crucial for complex data representation.
• Estimating the log-likelihood gradient is fundamental for training these models effectively.
• Existing methods often rely on approximations that lack theoretical guarantees.

Purpose of the Study:

• To analyze an expansion of the log-likelihood for undirected graphical models.
• To investigate estimators of the log-likelihood gradient derived from this expansion.
• To provide theoretical justification for approximations like Contrastive Divergence (CD).

Main Methods:

• Expansion of the log-likelihood function for undirected graphical models.
• Analysis of Gibbs chain sampling for estimating gradient terms.
• Theoretical convergence proofs for residual terms.
• Empirical studies on the bias and effectiveness of the Contrastive Divergence (CD) estimator.

Main Results:

• The residual term in the log-likelihood expansion converges to zero, validating truncated Gibbs chains.
• This truncation forms the basis of the Contrastive Divergence (CD) estimator for log-likelihood gradients.
• A connection is established between stochastic reconstruction error and mean-field approximations used in autoassociators.
• Theoretical and empirical evidence links the number of Gibbs steps (k) and RBM parameter magnitudes to CD estimator bias.

Conclusions:

• The study provides theoretical grounding for using short Gibbs chains in RBM training via Contrastive Divergence (CD).
• CD-k serves as a reliable descent direction even with significant bias for small k.
• The findings are generalizable beyond RBMs, applicable to any model requiring Gibbs chain convergence.