Multi-step first: A lightweight deep reinforcement learning strategy for robust continuous control with partial...

Area of Science:

• Robotics and Artificial Intelligence
• Machine Learning and Control Theory

Background:

• Deep Reinforcement Learning (DRL) excels in fully observable Markov Decision Processes (MDPs).
• Performance dynamics shift in Partially Observable MDPs (POMDPs) due to incomplete state information.
• Existing DRL benchmarks often focus on MDPs, leaving POMDP performance less understood.

Purpose of the Study:

• To empirically compare PPO, TD3, and SAC algorithms on POMDP variants of continuous-control tasks.
• To investigate the impact of partial observability on the relative performance of leading DRL algorithms.
• To identify algorithmic adaptations that improve robustness in POMDP settings.

Main Methods:

• Comparative analysis of Proximal Policy Optimization (PPO), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC).
• Evaluation on representative POMDP formulations of standard continuous-control benchmarks.
• Introduction of multi-step bootstrapping to PPO and multi-step targets to TD3 (MTD3) and SAC (MSAC).

Main Results:

• PPO demonstrated superior robustness and performance under partial observability, contrasting typical MDP results.
• TD3 and SAC showed reduced performance in POMDPs compared to their MDP counterparts.
• Modified TD3 (MTD3) and SAC (MSAC) with multi-step targets exhibited improved robustness in POMDPs.

Conclusions:

• Partial observability significantly impacts DRL algorithm performance, inverting typical rankings.
• PPO's inherent multi-step bootstrapping provides a stabilizing advantage in POMDPs.
• Adapting algorithms like TD3 and SAC with multi-step targets offers a practical method to enhance their robustness in partially observable environments.