Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Neuronal adaptation inspires new methods for understanding neural network dynamics.
• Existing normalization techniques face challenges with limited, imbalanced, or non-stationary data.
• Unsupervised attention mechanisms are crucial for interpreting complex deep learning models.

Purpose of the Study:

• To introduce Regularity Normalization (RN) as an Unsupervised Attention Mechanism (UAM).
• To leverage the Minimum Description Length (MDL) principle for statistical regularity computation.
• To enhance neural network robustness across diverse and challenging data distributions.

Main Methods:

• Developed RN to compute statistical regularity in neural network implicit spaces.
• Framed neural network optimization as a partially observable model selection problem.
• Utilized a normalization factor, the universal code length, computed incrementally across layers.

Main Results:

• RN outperforms existing normalization methods on tasks with limited, imbalanced, and non-stationary data.
• Demonstrated flexibility in incorporating data priors like top-down attention.
• Validated effectiveness across image classification, control, reinforcement learning, generative modeling, and NLP tasks.

Conclusions:

• Regularity Normalization offers a robust unsupervised attention mechanism for deep learning.
• RN enhances model performance and interpretability, particularly in challenging data scenarios.
• The UAM serves as a valuable tool for probing network dependencies and learning stages.