Expansion of Information in the Binary Autoencoder With Random Binary Weights

Area of Science:

• Computational neuroscience
• Information theory
• Machine learning

Background:

• Biological neural networks often feature expanded hidden layers, seen in sensory systems like the fruit fly's olfactory system and thalamic projections to the neocortex.
• Understanding information processing in such expanded networks is crucial for modeling neural computation.

Purpose of the Study:

• To investigate information expansion and preservation in binary autoencoders with larger hidden layers.
• To analyze the impact of encoding sparsity and dimension on reconstruction, similarity preservation, and mutual information.

Main Methods:

• Analysis of three models: threshold, k-winner-take-all (kWTA), and binary matching pursuit.
• Evaluation of information flow and reconstruction accuracy across layers under varying sparsity levels.

Main Results:

• Sparser hidden layer activation is beneficial for preserving information between input and output layers.
• Optimal similarity preservation occurs with dense hidden layer activation across all models.
• Zero reconstruction error is achievable with a sufficiently large hidden layer by adjusting neuron thresholds.
• Sparsity preference arises from noise in the weight matrix; fixed non-zero connections improve dense activation performance.

Conclusions:

• Sparse binary representations and association memory are key in neural computation models.
• The interplay between sparsity, density, and noise determines information processing efficiency in expanded neural networks.