Related Experiment Videos
Multiagent reinforcement learning in the Iterated Prisoner's Dilemma
1Computer Science Department, University of Massachusetts at Amherst 01003, USA.
Bio Systems
|January 1, 1996
Summary
Reinforcement learning agents using Q-learning can play the Iterated Prisoner's Dilemma. Agents with longer histories and memory performed best against adaptive opponents, showing improved reinforcement learning strategies.
Area of Science:
- Artificial Intelligence
- Game Theory
- Computational Neuroscience
Background:
- Reinforcement learning (RL) strengthens actions based on outcomes.
- Q-learning is an adaptable RL algorithm suitable for repeated games.
- Most RL research focuses on single-agent or zero-sum/team scenarios.
Purpose of the Study:
- Empirically study reinforcement learning agents in the Iterated Prisoner's Dilemma (IPD).
- Investigate Q-learning agent performance against fixed and adaptive opponents.
- Analyze factors influencing RL agent success in non-cooperative, non-zero-sum games.
Main Methods:
- Implemented various Q-learning agents with differing history lengths and memory types (lookup tables vs. recurrent neural networks).
- Tested agents against a fixed Tit-For-Tat strategy and other Q-learning agents.
- Varied exploration schedules for different learning agents.
Main Results:
- Q-learning agents achieved optimal play against the Tit-For-Tat strategy.
- Playing against adaptive Q-learning opponents proved more challenging due to non-stationary environments.
- Agents with longer history windows, lookup table memories, and extended exploration schedules demonstrated superior performance in IPD.
Conclusions:
- Reinforcement learning in complex domains like IPD presents significant challenges.
- Agent memory and exploration strategies are critical for success against adaptive opponents.
- Q-learning agents can learn effective strategies in multi-agent settings with non-correlated payoffs.