Investigation of independent reinforcement learning algorithms in multi-agent environments

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multi-Agent Systems

Background:

• Independent reinforcement learning (IRL) algorithms lack theoretical guarantees in multi-agent settings.
• Existing research shows varied practical performance of IRL algorithms across different domains.
• A comprehensive analysis of IRL algorithm strengths and weaknesses is needed.

Purpose of the Study:

• To empirically compare the performance of IRL algorithms across diverse multi-agent environments.
• To investigate the impact of environment type (cooperative, competitive, mixed) on IRL performance.
• To evaluate the effectiveness of adding recurrence to IRL algorithms in partially-observable settings.

Main Methods:

• Performance evaluation of IRL algorithms on seven PettingZoo environments.
• Categorization of environments into cooperative, competitive, and mixed settings.
• Comparison of IRL algorithms against multi-agent algorithms.

Main Results:

• IRL algorithms perform comparably to multi-agent algorithms in fully-observable cooperative environments.
• Recurrence enhances IRL algorithm learning in partially-observable cooperative settings.
• IRL algorithms achieve competitive or superior performance in competitive environments.
• IRL algorithms fail to learn effective cooperation and competition strategies in mixed environments.

Conclusions:

• IRL algorithms are viable for cooperative and competitive tasks, especially with architectural enhancements like recurrence.
• Significant limitations exist for IRL algorithms in mixed cooperative-competitive scenarios.
• Further research is needed to address the shortcomings of IRL in complex mixed-agent interactions.