Orientation-Preserving Rewards' Balancing in Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Auxiliary rewards are common in complex reinforcement learning (RL) but often interfere with the primary objective, degrading policy performance.
• Existing methods struggle to balance main and auxiliary rewards effectively, posing a significant challenge in RL research.

Purpose of the Study:

• To address the challenge of balancing main and auxiliary rewards in reinforcement learning.
• To develop a method that preserves the policy's optimization for main rewards while incorporating auxiliary rewards.
• To ensure the policy optimized with balanced rewards remains consistent with the policy optimized solely with main rewards.

Main Methods:

• Formulating reward balancing as a search for a Pareto optimal solution.
• Proposing a variant Pareto optimization method to guide policy search towards main rewards.
• Establishing an iterative learning framework for reward balancing with theoretical analysis of convergence and time complexity.

Main Results:

• The proposed Pareto optimization variant effectively guides policy search towards main rewards.
• Experimental results in discrete (grid world) and continuous (Doom) environments demonstrate effective reward balancing.
• The algorithm achieved remarkable performance compared to reinforcement learning methods with heuristic rewards, learning expert-level policies on the ViZDoom platform.

Conclusions:

• The proposed Pareto-based approach offers an effective solution for balancing main and auxiliary rewards in reinforcement learning.
• This method successfully preserves the optimization orientation towards main rewards, leading to improved policy performance.
• The iterative learning framework is theoretically sound and empirically validated, showing promise for complex RL applications.