An information-theoretic approach to curiosity-driven reinforcement learning

Area of Science:

• Artificial Intelligence
• Information Theory
• Machine Learning

Background:

• Reinforcement learning (RL) relies on exploration strategies to balance reward acquisition and learning.
• Boltzmann-style exploration is a common method, but its theoretical optimality is re-examined.
• Curiosity-driven learning aims to enhance agent engagement and knowledge acquisition.

Purpose of the Study:

• To provide an information-theoretic perspective on exploration in reinforcement learning.
• To demonstrate the optimality of Boltzmann-style exploration concerning expected return and policy coding cost.
• To introduce a novel framework for curiosity-driven learning that integrates predictive power maximization.

Main Methods:

• Information-theoretic analysis of exploration strategies.
• Derivation of optimal policies balancing expected return and information gain.
• Formulation of a new exploration-exploitation trade-off based on predictive power.

Main Results:

• Boltzmann-style exploration is shown to be information-theoretically optimal.
• A novel exploration bonus is introduced, enhancing curiosity-driven learning.
• The proposed exploration-exploitation trade-off is inherent in optimal deterministic policies.

Conclusions:

• Exploration in reinforcement learning can be viewed as optimizing information gain.
• Curiosity-driven learning enhances agent performance by maximizing predictive power.
• This approach offers a more principled understanding of exploration beyond random action selection.