t-soft update of target network for deep reinforcement learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Deep reinforcement learning (DRL) uses target networks to stabilize training by generating reference signals.
• Conventional exponential moving average update rules can propagate incorrect parameter updates, increasing learning variance.
• Existing methods to mitigate this risk often slow down the learning process.

Purpose of the Study:

• To introduce a novel, robust update rule for DRL target networks.
• To enhance learning stability and speed by addressing the limitations of conventional update methods.
• To develop a method that prevents erroneous parameter updates without sacrificing learning efficiency.

Main Methods:

• A new t-soft update rule is proposed, drawing inspiration from the Student-t distribution.
• The method leverages an analogy between exponential moving average and normal distribution properties.
• The t-soft update rule is analyzed to demonstrate its unique properties, including a heavy-tailed characteristic.

Main Results:

• The t-soft update rule exhibits properties of the Student-t distribution, notably a heavy-tailed characteristic.
• It automatically excludes extreme parameter updates that diverge from past experiences.
• The method accelerates learning when updates align with past data, outperforming conventional methods in PyBullet robotics simulations.

Conclusions:

• The t-soft update rule offers a robust alternative to conventional target network updates in DRL.
• It effectively balances the need for stability with the requirement for rapid learning.
• Empirical results in robotics simulations demonstrate superior performance in terms of return and reduced variance.