Dynamical systems as a level of cognitive analysis of multi-agent learning: Algorithmic foundations of...

Area of Science:

• Cognitive Science
• Machine Learning
• Game Theory

Background:

• Multi-agent learning dynamics are often analyzed using a dynamical systems perspective, linking evolutionary game theory and reinforcement learning.
• Existing interpretations of dynamical systems in multi-agent learning lack clarity, necessitating a more structured approach.

Purpose of the Study:

• To embed dynamical systems descriptions of multi-agent learning within different cognitive analysis levels.
• To clarify the connections between these abstraction levels for enhanced insight into multi-agent learning.
• To demonstrate the framework's utility with temporal-difference reinforcement learning.

Main Methods:

• Utilizing a dynamical systems framework to analyze multi-agent learning.
• Connecting evolutionary game theory and reinforcement learning.
• Proposing an on-line sample-batch temporal-difference algorithm with memory-batch and separated state-action value estimation.

Main Results:

• The deterministic dynamical systems description of temporal-difference reinforcement learning adheres to a minimum free-energy principle.
• This approach unifies boundedly rational game theory with decision-making under uncertainty.
• The proposed algorithm provides a micro-foundation for deterministic learning equations, with learning trajectories converging to deterministic equations under large batch sizes.

Conclusions:

• Embedding dynamical systems within cognitive analysis levels offers a robust framework for understanding multi-agent learning.
• The framework guides the full utilization of the dynamical systems approach in multi-agent learning research.
• This work bridges theoretical insights from game theory and practical applications in reinforcement learning.