Deep Deterministic Policy Gradient With Compatible Critic Network.

Area of Science:

• Reinforcement Learning
• Robotics
• Machine Learning

Background:

• Deep Deterministic Policy Gradient (DDPG) is effective for continuous control but faces challenges with critic network compatibility.
• Critic network compatibility ensures policy evaluation aligns with policy improvement, crucial for convergence but often restrictive.

Purpose of the Study:

• To develop concrete methods for measuring critic network compatibility in DDPG.
• To demonstrate the necessity of compatible critic networks and improve DDPG performance.

Main Methods:

• Introduced neural network similarity indices (e.g., Centered Kernel Alignment, Normalized Bures) using kernel matrices to quantify compatibility.
• Applied the sketching trick to reduce computational cost of similarity calculations.
• Remodeled the compatible function using an energy function model and incorporated policy change information.

Main Results:

• Empirically validated Centered Kernel Alignment and Normalized Bures similarity indices for consistent compatibility scores.
• Demonstrated the necessity of compatible critic networks through theoretical analysis and experiments.
• Achieved higher compatibility scores and improved performance by integrating policy change information and proposing a light-computation overestimation solution.

Conclusions:

• Critic network compatibility is essential for DDPG's convergence and performance.
• Novel similarity indices provide a practical way to assess and improve critic network compatibility.
• The proposed methods enhance DDPG's suitability for large-scale continuous control problems in robotics.