Risk sensitive twin distributional critics with a lambda lower confidence bound for continuous control reinforcement...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Off-policy actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), are foundational for continuous-control reinforcement learning.
• Existing methods lack explicit mechanisms for risk control in temporal-difference targets, relying on scalar value estimates.

Purpose of the Study:

• Introduce Twin Distributional Critics with λ-Lower Confidence Bound (TDC-λ), a novel algorithm for risk-sensitive reinforcement learning.
• Develop a TD3-style algorithm that incorporates distributional critics and a risk parameter (λ) for target selection.

Main Methods:

• Implemented a TD3-style algorithm featuring two distributional critics.
• Formulated targets using a lower confidence bound (μ - λσ) across critics to manage risk.
• Supported both deterministic and Gaussian policies, with deterministic mean action used for evaluation.

Main Results:

• TDC-λ matched or improved final returns on standard MuJoCo benchmarks (HalfCheetah-v4, Hopper-v4, Ant-v4, Walker2d-v4, Humanoid-v4).
• Consistently reduced variance across tasks compared to strong baselines.
• Demonstrated improved robustness on high-dimensional domains with increased risk penalties (higher λ).

Conclusions:

• Distributional critics combined with risk-sensitive target selection significantly enhance stability in off-policy reinforcement learning.
• TDC-λ offers a method to improve robustness and reduce variance without compromising sample efficiency.