State-Temporal Compression in Reinforcement Learning With the Reward-Restricted Geodesic Metric

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) faces challenges with large state spaces and long-term decision-making.
• Compressing RL problems into smaller, manageable ones is a key strategy to overcome these difficulties.
• Compression must be state-temporal and preserve optimality for effective problem-solving.

Purpose of the Study:

• To propose a novel metric for state-temporal compression in reinforcement learning.
• To develop a reinforcement learning algorithm utilizing this compression for complex task solving.
• To demonstrate the effectiveness of the proposed method in diverse environments.

Main Methods:

• Introduced a reward-restricted geodesic (RRG) metric, learnable via neural networks, for state-temporal compression.
• Proved that RRG metric-based compression is approximately optimality-preserving for RL problems with temporally abstract actions.
• Designed and implemented an RRG metric-based reinforcement learning (RRG-RL) algorithm.

Main Results:

• The RRG metric enables effective state-temporal compression in reinforcement learning.
• The RRG-RL algorithm demonstrated superior performance compared to existing RL approaches.
• Experiments in discrete (2D Minecraft) and continuous (Doom) environments validated the method's efficacy.

Conclusions:

• The proposed RRG metric offers a promising approach for state-temporal compression in reinforcement learning.
• RRG-based compression facilitates the development of more efficient RL algorithms for complex tasks.
• This method advances the capability of reinforcement learning in handling large-scale and long-horizon problems.