Higher-Order Markov Model-Based Analysis of Reinforcement Learning in 6G Mobile Retrial Queueing Systems

Area of Science:

• Telecommunications Engineering
• Artificial Intelligence
• Queueing Theory

Background:

• 6G mobile communication services face dynamic challenges in queueing systems.
• Deep Q-Network Reinforcement Learning (DQN-RL) offers a potential solution for optimizing network behavior.
• Understanding agent learning convergence is crucial for effective implementation.

Purpose of the Study:

• To analyze the learning convergence of DQN-RL agents in 6G retrial queueing systems.
• To quantify convergence characteristics using Markov chain methods and mixing time analysis.
• To provide a foundation for optimizing 6G queueing strategies under uncertainty.

Main Methods:

• Utilized first- and second-order Markov chain methods to analyze DQN-RL agent convergence.
• Simulated temporal evolution of reward sequences as Markov chains.
• Employed mixing time analysis and spectral gap properties of Markov models to assess convergence.

Main Results:

• Markov chain analysis indicates 10 training episodes are sufficient for policy convergence in DQN-RL.
• In some scenarios, as few as 5 episodes enhance mobile network performance with low energy consumption.
• Mixing time calculations assessed learning stability and system responsiveness across 120 parameter combinations.

Conclusions:

• DQN-RL convergence is sensitive to system parameters and retrial dynamics in 6G queueing.
• Markov chain analysis provides a rigorous method for evaluating learning convergence.
• The findings support optimizing 6G queueing strategies for enhanced efficiency and performance.