Dataset-Learning Duality and Emergent Criticality.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Artificial neural networks (ANNs) involve complex interactions between trainable variables (weights, biases) and non-trainable variables (activations, data).
• Understanding the relationship between data properties and learning dynamics is crucial for optimizing ANN performance and interpretability.

Purpose of the Study:

• To establish and explore a mathematical duality between datasets and the learning dynamics of trainable variables in ANNs.
• To investigate the emergence of criticality (power-law distributions) in ANN learning using this duality.
• To analyze how activation and loss functions influence learning criticality.

Main Methods:

• Formulated a "dataset-learning duality" by composing activation and learning passes (e.g., forward and backward propagation).
• Analyzed this duality in both toy and large-scale ANN models at learning equilibrium.
• Studied the impact of modifying activation and loss functions on the emergent power-law distributions.

Main Results:

• Demonstrated a complex nonlinear duality map between non-trainable boundary variables (dataset) and trainable variables (learning).
• Showed that criticality can emerge in the learning system even when the dataset is in a non-critical state.
• Confirmed that the power-law distribution of trainable variable fluctuations can be altered by changing activation or loss functions.

Conclusions:

• The dataset-learning duality provides a novel framework for understanding ANN learning dynamics and emergent properties.
• Criticality in ANNs is not solely determined by the dataset but is also shaped by network architecture and learning rules.
• This duality offers insights into controlling and predicting learning behaviors in complex neural systems.