Heavy-tailed update distributions arise from information-driven self-organization in nonequilibrium learning

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Artificial neural networks (ANNs) often balance exploration in parameter space with task-specific adaptation, mirroring human decision-making.
• Understanding the dynamics of ANNs during training is crucial for improving AI performance and interpretability.

Purpose of the Study:

• To identify consistent signatures of criticality during neural network training.
• To provide theoretical evidence for information-driven self-organization as the mechanism behind observed scaling behavior.
• To investigate the intrinsic geometric properties of the loss landscape and the nature of the learning process.

Main Methods:

• Identifying consistent signatures of criticality during neural network training.
• Developing theoretical models based on maximum entropy and mutual information principles.
• Numerical simulations to demonstrate self-organized criticality and analyze loss landscape properties.
• Analyzing power-law distributions in parameter updates and inter-update intervals.

Main Results:

• Consistent signatures of criticality were identified during neural network training.
• Theoretical evidence suggests criticality arises from a balance between maximum entropy (exploration) and mutual information (task relevance).
• The loss landscape shows a transition from exponential to power-law ruggedness, indicating an intrinsic geometric property.
• A power-law distribution in update intervals suggests an intermittent learning process.

Conclusions:

• Neural network learning is a nonequilibrium process governed by a trade-off between randomness and relevance.
• Self-organized criticality provides a framework for understanding the dynamic and adaptive nature of ANNs.
• Findings offer insights into the interpretability of artificial intelligence systems.