Modeling and Planning with Macro-Actions in Decentralized POMDPs.

Area of Science:

• Artificial Intelligence
• Robotics
• Decision Theory

Background:

• Decentralized partially observable Markov decision processes (Dec-POMDPs) are standard for multi-agent decision-making under uncertainty.
• Traditional Dec-POMDPs use primitive, single-time-step actions, limiting scalability and temporal abstraction.
• Handling temporally extended actions (macro-actions) in decentralized systems presents significant challenges due to asynchronous termination.

Purpose of the Study:

• To extend existing Dec-POMDP algorithms to effectively model and solve problems involving macro-actions (options).
• To enable decentralized agents to coordinate using temporally extended actions, improving scalability and solution quality.
• To demonstrate the efficacy of the proposed approach in complex benchmarks and multi-robot coordination tasks.

Main Methods:

• Modeling macro-actions as options within the Dec-POMDP framework, focusing on agent-specific information during execution.
• Extending three leading Dec-POMDP policy generation algorithms to accommodate the asynchronous nature of options.
• Evaluating the extended algorithms on standard benchmarks and a multi-robot coordination problem.

Main Results:

• The developed algorithms successfully retain agent coordination capabilities with macro-actions.
• High-quality solutions were generated for significantly longer horizons and larger state-spaces compared to prior Dec-POMDP methods.
• The approach demonstrated effectiveness in multi-robot coordination, balancing uncertainty and sensor information.

Conclusions:

• Macro-actions (options) can be effectively integrated into Dec-POMDPs to enhance decision-making capabilities for decentralized agents.
• The extended algorithms offer a more scalable and versatile approach for complex multi-agent coordination problems.
• This work provides a unified framework for synthesizing control policies that leverage coordination opportunities in uncertain environments.