A Decomposition Optimization-Based Multiobjective Reinforcement Learning Algorithm for Obtaining Nonconvex Pareto...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization

Background:

• Multiobjective reinforcement learning (MORL) seeks Pareto fronts (PFs) in multiobjective Markov decision processes (MOMDPs).
• Existing MORL algorithms struggle with nonconvex PFs, limiting their applicability.
• This limitation hinders the discovery of diverse, optimal policies in complex scenarios.

Purpose of the Study:

• To propose a novel nonlinear MORL algorithm, MORL/D-VR, capable of handling nonconvex PFs.
• To provide a theoretical guarantee for finding Pareto optimal policies irrespective of PF shape.
• To enhance policy gradient methods for improved performance and diversity.

Main Methods:

• Decomposition of MOMDPs into single-objective MDPs using the Tchebycheff approach.
• Application of an improved policy gradient algorithm, expected utility policy gradient (EUPG).
• Implementation of variance reduction techniques and weight vector adaptation for enhanced performance.

Main Results:

• MORL/D-VR demonstrates theoretical Pareto optimality for nonconvex PFs.
• The algorithm achieves desirable performance on both convex and nonconvex PF problems.
• Experimental results show MORL/D-VR outperforms current state-of-the-art MORL algorithms.

Conclusions:

• MORL/D-VR effectively overcomes limitations of existing MORL algorithms in handling nonconvex PFs.
• The proposed method offers a theoretical foundation for achieving Pareto optimality in complex MOMDPs.
• MORL/D-VR represents a significant advancement in MORL, improving policy discovery and performance.