Distributed Q-Learning Algorithm for Dynamic Resource Allocation With Unknown Objective Functions and Application to...

Area of Science:

• Distributed Artificial Intelligence
• Reinforcement Learning
• Optimization

Background:

• The dynamic resource allocation problem (DRAP) involves optimizing resource distribution over time with uncertain cost and transition functions.
• Decentralized decision-making among agents is essential, requiring communication only with neighbors.

Purpose of the Study:

• To develop distributed Q-learning algorithms for DRAP under both discrete and continuous local feasibility constraints.
• To ensure that the proposed algorithms guarantee feasible resource allocations at each time step.

Main Methods:

• A distributed Q-learning algorithm is proposed for DRAP with discrete local feasibility constraints (DLFCs).
• A novel distributed Q-learning algorithm utilizing function approximation and distributed optimization is developed for continuous local feasibility constraints (CLFCs).

Main Results:

• Theoretical proof confirms that the distributed Q-learning algorithm for DLFCs always yields a feasible allocation (FA).
• The proposed algorithm for CLFCs also ensures that the joint policy of agents constitutes an FA at each time period.
• Simulations demonstrate the effectiveness of both proposed algorithms.

Conclusions:

• The developed distributed Q-learning algorithms effectively address DRAP with unknown cost and transition functions.
• The guarantee of feasible allocations is critical for the successful execution of learning policies, such as epsilon-greedy.
• The research provides robust solutions for decentralized resource management in dynamic environments.