Information-theoretic analysis of Hierarchical Temporal Memory-Spatial Pooler algorithm with a new upper bound for...

Area of Science:

• Machine Learning
• Computational Neuroscience
• Information Theory

Background:

• Hierarchical Temporal Memory (HTM) is an unsupervised machine learning algorithm inspired by neocortical principles.
• The Spatial Pooler (SP) is a core HTM component that generates sparse distributed representations from binary input streams.
• Evaluating SP's sparsification using information theory metrics is crucial for understanding its performance.

Purpose of the Study:

• To assess the sparsification effectiveness of the Spatial Pooler (SP) algorithm within Hierarchical Temporal Memory (HTM).
• To introduce and utilize a modified information bottleneck (IB) measure to evaluate SP performance under varying sparsity and noise conditions.
• To mathematically prove the positive correlation between sparsity and SP algorithm performance.

Main Methods:

• Information-theoretic analysis using the information bottleneck (IB), Cramer-Rao lower bound, and Fisher information matrix.
• Introduction of a novel upper bound for the IB relation, termed modified-IB.
• Testing the SP algorithm with MNIST, Fashion-MNIST, and NYC-Taxi datasets, including introducing up to 40% noise.
• Reconstruction of sparse SP output representations using probabilistic mapping and Hidden Markov Models.
• Mathematical analysis of the Cramer-Rao lower bound with Cauchy distributed data.

Main Results:

• The SP algorithm demonstrated significant resistance to noise, with up to 40% input noise causing no discernible output changes.
• Numerical calculations using modified-IB indicated that lower noise and higher sparsity levels improve reconstruction effectiveness.
• The SP algorithm achieved optimal results with 2% sparsity.
• Mathematical proof confirmed that increased sparsity enhances SP algorithm performance.

Conclusions:

• The Spatial Pooler (SP) algorithm exhibits robust performance and noise resistance, particularly at higher sparsity levels.
• The modified-IB measure provides a valuable tool for evaluating SP performance across different sparsity and noise conditions.
• Sparsity is a critical factor for optimizing the SP algorithm's efficiency and representational capabilities in Hierarchical Temporal Memory.