The Hierarchical Continuous Pursuit Learning Automation: A Novel Scheme for Environments With Large Numbers of...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Traditional learning automata (LA) methods struggle with environments featuring a large number of actions due to high-dimensional action probability vectors.
• Very large action spaces lead to action probabilities becoming smaller than machine accuracy, preventing their selection and hindering effective learning.

Purpose of the Study:

• To present a novel hierarchical solution extending the continuous pursuit paradigm to learning automata problems with a large number of actions.
• To address the challenge of arbitrarily small action probabilities in high-dimensional action spaces.

Main Methods:

• A hierarchical structure is proposed where all environment actions are leaves.
• Each level of the hierarchy utilizes a two-action learning automaton.
• The pursuit paradigm is employed at each level, allowing the best action to propagate upwards towards the root.

Main Results:

• Formal proof of ϵ-optimal convergence for the hierarchical scheme.
• Extensive experimental validation on environments with up to 256 actions.
• Demonstrated computational advantages, with the hierarchical continuous pursuit automaton requiring up to 82% fewer iterations than the benchmark L_R-I scheme.

Conclusions:

• The proposed hierarchical learning automata scheme effectively solves problems with large action spaces.
• The method offers significant computational advantages and converges to optimal actions.
• The approach is extendable to discretized and Bayesian pursuit learning automata.