Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Ensuring safety in offline reinforcement learning (RL) is crucial for safety-critical applications.
• Optimizing distributional value functions in offline RL faces challenges like quantile crossing and distribution shift.

Purpose of the Study:

• To develop a risk-averse policy learning method for offline RL.
• To address challenges in learning distributional value functions and ensure policy safety.

Main Methods:

• Proposed a monotonic quantile network (MQN) for learning return distributions with non-crossing quantiles.
• Implemented conservative quantile regression (CQR) to penalize out-of-distribution actions.
• Learned a worst-case policy by optimizing conditional value-at-risk (CVaR).

Main Results:

• The proposed method, MQN with CQR, effectively learns safe and conservative policies.
• Experimental results in robotic locomotion tasks demonstrate the method's efficacy in both risk-neutral and risk-sensitive settings.
• Theoretical analysis confirmed fixed-point convergence of the method.

Conclusions:

• The MQN with CQR approach successfully enables safe and risk-averse policy learning in offline RL.
• This method is particularly valuable for safety-critical domains requiring conservative decision-making.
• The non-crossing quantile guarantee and OOD action penalty are key to achieving robust and safe policies.