Complexity and non-commutativity of learning operations on graphs

Area of Science:

• Computational neuroscience
• Machine learning theory
• Complex systems

Background:

• Recurrent neural networks (RNNs) are fundamental to processing sequential data.
• Understanding the internal dynamics and learning mechanisms of small RNNs is crucial for developing more efficient AI.
• Graph theory provides a powerful framework for analyzing network structures and their emergent properties.

Purpose of the Study:

• To investigate supervised learning in small recurrent networks by modeling them as graphs.
• To characterize the stability and representational structure of optimized networks.
• To explore the relationship between learning complexity and network dynamics.

Main Methods:

• Numerical simulations of supervised learning tasks in small recurrent networks.
• Analysis of network dynamics using graph theory, focusing on attractors and stability.
• Investigation of the non-commutative properties of input-attractor mappings.
• Examination of the non-monotonic behavior of attractor set size during learning.

Main Results:

• Optimized recurrent networks exhibit asymptotic stability characterized by attractors.
• These attractors form a representation space for associative multiplicative input operations.
• The mapping of input sequences to attractors is generally non-commutative.
• The complexity of learning, measured by the number of attractors, displays non-monotonic behavior.

Conclusions:

• Small recurrent networks, when optimized for supervised learning, develop stable, structured representations.
• The non-commutative nature of these representations reflects the sequential dependency of learning.
• The non-monotonic complexity suggests intricate dynamics in the learning process.
• A potential link between learning complexity and pragmatic information is suggested.