Deep Reinforcement Learning Data Collection for Bayesian Inference of Hidden Markov Models

Area of Science:

• Dynamical systems modeling
• Machine learning

Background:

• Hidden Markov Models (HMMs) are crucial for analyzing complex, partially observed systems.
• Current data collection for HMMs is often inefficient, especially with costly data in stochastic domains.

Purpose of the Study:

• To introduce a novel Bayesian lookahead data collection method for HMM inference.
• To optimize data collection strategies under uncertainty for improved long-term model performance.

Main Methods:

• Developed a Bayesian lookahead policy using a belief state to capture joint distributions of states and models.
• Employed deep reinforcement learning to approximate the optimal Bayesian solution via offline trajectory simulation.
• Created a pre-trained policy adaptable for real-time execution and dynamic adjustments.

Main Results:

• Demonstrated significant improvements in inference accuracy and robustness across three distinct systems.
• Showcased the method's effectiveness in data-limited and uncertain environments.
• Validated the approach's ability to support diverse inference objectives (point, distribution, causal).

Conclusions:

• The proposed Bayesian lookahead method offers a more efficient and robust approach to data collection for HMM inference.
• This framework enhances model performance by considering the long-term impact of data collection decisions.
• The deep reinforcement learning-based policy provides a practical and adaptive solution for real-world applications.