The Potential Energy of an Autoencoder

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Autoencoders are widely used for feature learning due to their simplicity and efficiency.
• Existing methods often heuristically infer autoencoder energy landscapes by linking them to probabilistic models like Restricted Boltzmann Machines (RBMs).

Purpose of the Study:

• To demonstrate that common autoencoders are inherently associated with an energy function, independent of training.
• To analytically derive the energy landscape by integrating the autoencoder's reconstruction function.
• To elucidate the relationship between autoencoders and RBMs.

Main Methods:

• Analytical derivation of the energy function for autoencoders.
• Integration of the autoencoder's reconstruction function to infer the energy landscape.
• Comparison of the derived autoencoder energy function with the free energy of RBMs.

Main Results:

• Most common autoencoders possess a natural energy function, enabling analytical energy landscape inference.
• For sigmoid hidden units, the autoencoder energy function is identical to the RBM free energy.
• The autoencoder energy function provides a dynamical systems perspective on regularization techniques like contractive training.

Conclusions:

• Autoencoders have an intrinsic energy function, simplifying their analysis and connection to probabilistic models.
• The analytical energy function deepens the understanding of autoencoder-RBM relationships.
• The energy function facilitates novel applications, such as generative classification using class-specific autoencoders.