Adaptive internal state space construction method for reinforcement learning of a real-world agent

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Reinforcement learning in real-world applications faces challenges in discretizing continuous sensory data.
• Traditional grid-based state-space discretization suffers from the curse of dimensionality.
• Global basis functions in continuous function approximators may not guarantee convergence.

Purpose of the Study:

• To propose a novel method for constructing discrete state spaces from continuous sensory input for reinforcement learning.
• To address the limitations of existing state-space discretization techniques.

Main Methods:

• Introduced the adaptive basis division (ABD) algorithm, which incrementally assigns local basis functions based on task requirements.
• Utilized the statistical properties of locally weighted temporal difference (TD) error to divide basis functions.
• Applied the ABD algorithm to simulated autonomous robot collision avoidance and real mobile robot navigation tasks.

Main Results:

• The ABD algorithm achieved collision avoidance with fewer basis functions compared to conventional methods in simulations.
• A real mobile robot successfully navigated to a goal using learned action strategies with fewer internal states than conventional methods.
• Demonstrated the efficiency and effectiveness of the proposed adaptive basis division approach.

Conclusions:

• The adaptive basis division algorithm offers an efficient solution for discretizing continuous sensory input in reinforcement learning.
• ABD reduces the number of required basis functions and internal states, outperforming conventional methods in practical robotic applications.