Information Dropout: Learning Optimal Representations Through Noisy Computation.

Area of Science:

• Deep Learning
• Information Theory
• Representation Learning

Background:

• Cross-entropy loss is common in deep learning but lacks key properties for optimal representations.
• Regularization can improve representation learning, with methods like dropout being a special case.

Purpose of the Study:

• To introduce a regularized loss function that enforces properties of optimal representations.
• To link representation learning, information theory, and variational inference.
• To promote the creation of optimal disentangled representations.

Main Methods:

• Adding a regularization term to cross-entropy loss, related to multiplicative noise injection.
• Utilizing Information Dropout, a generalization of dropout based on information-theoretic principles.
• Enforcing a factorized prior to promote disentangled representations.

Main Results:

• The regularized loss function can be efficiently minimized using Information Dropout.
• Information Dropout adapts noise to data and network capacity, outperforming binary dropout on smaller models.
• The proposed loss function yields a Variational Autoencoder in reconstruction tasks.

Conclusions:

• Information Dropout offers a principled way to improve deep learning representations and generalization.
• The method connects representation learning with information theory and variational inference.
• Enforcing factorized priors is a viable strategy for learning disentangled representations.