Learn Quasi-Stationary Distributions of Finite State Markov Chain

Area of Science:

• Computational Statistics
• Machine Learning
• Probability Theory

Background:

• Quasi-stationary distributions are crucial for analyzing systems that eventually reach a steady state.
• Traditional methods for computing these distributions can be computationally intensive and complex.
• Fixed-point formulations offer a theoretical basis but require efficient solution techniques.

Purpose of the Study:

• To develop a reinforcement learning (RL) based approach for computing quasi-stationary distributions.
• To address the challenge of minimizing KL-divergence between Markovian path distributions.
• To provide a practical algorithm for learning optimal solutions and value functions.

Main Methods:

• Formulating the quasi-stationary distribution problem as a fixed-point minimization.
• Employing reinforcement learning by defining reward and value functions.
• Deriving a policy gradient theorem tailored for this problem.
• Implementing an actor-critic algorithm for optimization.

Main Results:

• Successfully applied RL to compute quasi-stationary distributions.
• Demonstrated the efficacy of the actor-critic algorithm in learning optimal policies.
• Validated the method through numerical examples on finite state Markov chains.

Conclusions:

• The proposed RL approach offers an effective alternative for computing quasi-stationary distributions.
• The actor-critic algorithm provides a robust framework for solving KL-divergence minimization problems in this context.
• This method shows promise for analyzing complex stochastic systems.