Reliability of internal prediction/estimation and its application. I. Adaptive action selection reflecting...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Model-free reinforcement learning systems often struggle with adaptive action selection.
• Existing methods may not optimally balance exploration and exploitation based on prediction accuracy.

Purpose of the Study:

• To propose an adaptive action-selection method for model-free reinforcement learning.
• To introduce a Reliability Index (RI) to estimate the accuracy of internal predictions.
• To utilize the RI for adaptive control of exploration and learning rates.

Main Methods:

• Developed a Reliability Index (RI) for value function estimation in temporal difference learning.
• Integrated the RI into the Boltzmann action-selection rule, replacing the temperature parameter.
• Applied the RI to adjust the learning coefficient, establishing it as a basis for meta-learning.

Main Results:

• The RI-based Q-learning system showed adaptive behavior with an adaptive learning coefficient and RI-discount rate.
• Statistical tests indicated increased exploration in the initial learning phase and accelerated learning thereafter.
• The proposed method was found to be less effective with actor-critic models.

Conclusions:

• The Reliability Index offers a novel approach to adaptive action selection in reinforcement learning.
• This method enhances learning efficiency by dynamically adjusting exploration and learning rates.
• Further research is needed to explore its applicability beyond Q-learning and address limitations.