Actor-Critic Learning Control With Regularization and Feature Selection in Policy Gradient Estimation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Actor-critic (AC) architectures are crucial for reinforcement learning (RL) with continuous states and actions.
• Improving learning efficiency and convergence in AC methods remains a key challenge, often addressed through regularization and feature learning in policy evaluation.

Purpose of the Study:

• To propose a novel actor-critic learning control method incorporating regularization and feature selection for enhanced policy gradient estimation.
• To introduce the Regularized Dual-Averaging policy gradient (RDA-PG) algorithm for efficient and effective RL.

Main Methods:

• Utilizing L1-regularization within the actor network to perform automatic feature selection.
• Employing the regularized dual-averaging (RDA) technique for policy parameter updates, balancing past policy gradients with L1-regularization.
• Establishing convergence guarantees using two-timescale stochastic approximation theory.

Main Results:

• RDA-PG successfully performs feature selection in the actor network, learning sparse representations for both stochastic and deterministic policies.
• The algorithm efficiently solves the minimization problem inherent in the RDA technique.
• Demonstrated superior performance compared to existing AC algorithms on RL benchmarks with irrelevant or redundant features.

Conclusions:

• The proposed RDA-PG algorithm offers an effective approach to enhance actor-critic learning through integrated feature selection and regularization.
• RDA-PG provides a robust method for learning near-optimal stochastic and deterministic policies, particularly in scenarios with high-dimensional or noisy feature spaces.