Mean-field message-passing equations in the Hopfield model and its generalizations

Area of Science:

• Computational Neuroscience
• Machine Learning
• Statistical Physics

Background:

• Restricted Boltzmann machines (RBMs) are increasingly used for deep neural network preprocessing.
• Hopfield networks are well-understood models for studying neural network dynamics.

Purpose of the Study:

• To analyze mean-field equations (belief propagation and Thouless-Anderson-Palmer [TAP]) within the Hopfield model.
• To demonstrate their application as fast, iterative message-passing algorithms for computing local neuron polarizations.
• To compare belief propagation and TAP equations for RBM learning.

Main Methods:

• Revisiting mean-field equations (belief propagation and TAP) in the context of Hopfield networks.
• Developing iterative message-passing algorithms for neuron polarization.
• Analyzing equation behavior in the retrieval phase for memorized patterns.
• Investigating modifications to TAP equations for correlated patterns.

Main Results:

• Belief propagation equations are pattern-dependent, while TAP equations are unique, making TAP superior for RBM learning.
• A modified TAP equation approach is proposed for correlated patterns in Hopfield models.
• This modification relates to message passing on multi-layered graphical models, essential for general RBMs.

Conclusions:

• TAP equations offer a more efficient and applicable method for RBM preprocessing compared to belief propagation.
• Correlated patterns necessitate modifications to TAP equations, revealing an underlying layered graphical structure relevant to advanced RBMs.