One-Shot Averaging for Distributed TD(λ) Under Markov Sampling

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Computing

Background:

• Reinforcement learning (RL) is crucial for sequential decision-making.
• Policy evaluation is a fundamental task in RL.
• Distributed setups offer potential for faster computation but face communication challenges.

Purpose of the Study:

• To investigate distributed policy evaluation using TD(λ) methods.
• To achieve linear speedup in distributed RL settings.
• To reduce communication overhead in distributed policy evaluation.

Main Methods:

• A distributed setup where each agent has a copy of the Markov Decision Process.
• Independent transition sampling by each agent.
• TD(λ) algorithm for policy evaluation.
• A novel "one shot averaging" procedure for aggregating agent results.

Main Results:

• Achieved a linear speedup for TD(λ) policy evaluation in a distributed setting.
• Demonstrated that N agents can evaluate a policy N times faster.
• Showed that the linear speedup is achievable when the target accuracy is small enough.
• The "one shot averaging" method significantly reduces communication requirements.

Conclusions:

• Distributed reinforcement learning with independent sampling and "one shot averaging" enables efficient policy evaluation.
• Linear speedup is attainable with reduced communication, outperforming previous distributed approaches.
• This method offers a practical approach for large-scale policy evaluation in RL.